Jekyll2024-02-06T04:05:39+00:00https://anonoz.github.io/feed.xmlAnonoz BurpsRails & DevOps fella from KL, Malaysia.How to retrieve KLSE, SGX, and gold prices in Google Sheets (Updated for 2022Q1)2022-02-28T00:00:00+00:002022-02-28T00:00:00+00:00https://anonoz.github.io/finance/2022/02/28/google-sheets-klse-sgx<p>For Bursa Malaysia (KLSE) and Singapore Exchange (SGX), we can scrape from <a href="https://klse.i3investor.com/index.jsp">i3investor</a> websites. This is how.</p>
<h3 id="klse--sgx">KLSE & SGX</h3>
<p>This is the new formula for KLSE after the site revamp:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>=index(ImportXML("https://klse.i3investor.com/web/stock/overview/5176?randomstring", "//div[contains(@id, 'stock-price-info')]//p[2]"), 1, 1)
</code></pre></div></div>
<p>And this is for SGX:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>=index(ImportXML("https://sgx.i3investor.com/servlets/stk/g3b.jsp", "//td[contains(@class, 'big16')]"), 1, 1)
</code></pre></div></div>
<p>Just remember to replace the stock code with whatever you want, the part right before <code class="language-plaintext highlighter-rouge">.jsp</code>.</p>
<h3 id="gold-prices">Gold Prices</h3>
<p>You can use this formula to approximate USD per oz:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>=GOOGLEFINANCE("GLD", "price")*10/POW(0.996,YEARFRAC(TODAY(), "18/11/2004", 1))
</code></pre></div></div>
<p>GLD is the ticker for SPDR gold trust ETF, the largest one out there. They charge 0.4% per annum since inception.</p>For Bursa Malaysia (KLSE) and Singapore Exchange (SGX), we can scrape from i3investor websites. This is how.How to install Svelte and Tailwindcss in Rails with jsbundling and cssbundling2021-12-11T00:00:00+00:002021-12-11T00:00:00+00:00https://anonoz.github.io/tech/2021/12/11/rails-tailwindcss-svelte<p>As of the publication date of this guide, Rails 7 is about to be released and with it, comes the new cssbundling-rails and jsbundling-rails gems from the Rails core team.</p>
<p><a href="https://github.com/rails/cssbundling-rails">cssbundling-rails</a> allows us to easily use other CSS transpilers such as Tailwind, PostCSS, DartSass apart from what is offered in Ruby gems.</p>
<p><a href="https://github.com/rails/jsbundling-rails">jsbundling-rails</a> allows us to use JS compilers other than webpack - which is absolutely painful to work with.</p>
<p>In this short tutorial, I will be using esbuild, which is easier to configure than webpack for those who only seek to build js files and not replace the whole Sprockets asset pipeline.</p>
<p>This short guide will only cover Svelte and Tailwind, because these are the tools we use in Talenox.</p>
<p>You will need these installed before you proceed: node, yarn, foreman.</p>
<h3 id="demo-codes">Demo codes</h3>
<p>I will put the demo codes on <a href="https://github.com/anonoz/demo-rails6-tailwind-svelte/commits/main">anonoz/demo-rails6-tailwind-svelte</a> repo. You are free to check the commit logs as you read along, clone it, and play with it. I have removed activerecord, activestorage, actionmailer so there is nothing much to setup.</p>
<p>You can create a simple page to test out the different CSS</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><div</span> <span class="na">class=</span><span class="s">"existing-css-file"</span><span class="nt">></span>
<span class="nt"><h1></span>This is old school sprockets css.<span class="nt"></h1></span>
<span class="nt"></div></span>
<span class="nt"><div</span> <span class="na">class=</span><span class="s">"container mx-auto"</span><span class="nt">></span>
<span class="nt"><h1</span> <span class="na">class=</span><span class="s">"text-3xl text-pink-900"</span><span class="nt">></span>This is Tailwind.<span class="nt"></h1></span>
<span class="nt"></div></span>
<span class="nt"><div</span> <span class="na">data-svelte-component=</span><span class="s">"DemoSvelteComponent"</span><span class="nt">></span>
<span class="nt"></div></span>
</code></pre></div></div>
<p>Add append the following into <code class="language-plaintext highlighter-rouge">app/assets/stylesheets/application.css</code></p>
<div class="language-css highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">.existing-css-file</span> <span class="nt">h1</span> <span class="p">{</span>
<span class="nl">font-size</span><span class="p">:</span> <span class="m">5rem</span><span class="p">;</span>
<span class="nl">color</span><span class="p">:</span> <span class="m">#324343</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Since we have not added Tailwindcss yet, we still have the original browser styles. Over the next few steps we will see how the web page’s looks change.</p>
<p><img src="/assets/images/rails-tailwind-svelte/step0.png" alt="Screenshot at the end of Step 0" /></p>
<h3 id="step-1-add-the-gems-and-run-their-setup-script">Step 1: Add the gems and run their setup script</h3>
<p>Follow their readmes and add these to your <code class="language-plaintext highlighter-rouge">Gemfile</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gem 'jsbundling-rails'
gem 'cssbundling-rails'
</code></pre></div></div>
<p>Then run:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bundle
bundle exec rails css:install:tailwind
</code></pre></div></div>
<p>After this step, you can use <code class="language-plaintext highlighter-rouge">bin/dev</code> to boot up Rails development server and Tailwindcss one shot. You may need to <code class="language-plaintext highlighter-rouge">gem install foreman</code> to start them with <code class="language-plaintext highlighter-rouge">Procfile.dev</code>.</p>
<p>Before you proceed, please check the files changed and make sure it did not wipe your codes out. You may want to rename the newly generated js and css files to avoid conflicts with your existing ones.</p>
<p><img src="/assets/images/rails-tailwind-svelte/step1.png" alt="Screenshot at the end of Step 1" /></p>
<p>As you can see in the screenshot above, the custom style for the first H1 is missing - the font size and color are gone.</p>
<h3 id="step-1a-recover-existing-applicationcss">Step 1a: Recover existing application.css</h3>
<p>As of now, running <code class="language-plaintext highlighter-rouge">rails css:install:tailwind</code> will delete your existing <code class="language-plaintext highlighter-rouge">app/assets/stylesheets/application.css</code>. Here is how to recover them:</p>
<p>First rename the output filename for tailwind. Inside <code class="language-plaintext highlighter-rouge">package.json</code>, change the filename arguments for tailwindcss:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>"scripts": {
"build:css": "tailwindcss -i ./app/assets/stylesheets/application.tailwind.css -o ./app/assets/builds/application.tailwind.css"
}
</code></pre></div></div>
<p>Update your layout file to reference the new CSS file</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code><!-- The original sprockets one -->
<%= stylesheet_link_tag 'application', media: 'all', 'data-turbolinks-track': 'reload' %>
<!-- The tailwind
<%= stylesheet_link_tag "application.tailwind" %>
</code></pre></div></div>
<p>After that, checkout the previous <code class="language-plaintext highlighter-rouge">application.css</code> from your Git.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git checkout -- app/assets/stylesheets/application.css
</code></pre></div></div>
<p>In <code class="language-plaintext highlighter-rouge">app/assets/config/manifest.js</code>, add back your application.css manually:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>//= link_tree ../images
//= link_tree ../builds
//= link application.css
</code></pre></div></div>
<p>If you have run <code class="language-plaintext highlighter-rouge">bin/dev</code> rails server before this step, you may need to clear your <code class="language-plaintext highlighter-rouge">app/assets/builds/</code> directory because there would be a Tailwind generated <code class="language-plaintext highlighter-rouge">application.css</code> inside. The new bundling gems do not clear the directories on every reload.</p>
<p>Now your old styling should be back:</p>
<p><img src="/assets/images/rails-tailwind-svelte/step1a.png" alt="Screenshot at the end of Step 1a" /></p>
<h3 id="step-2-add-svelte">Step 2: Add Svelte</h3>
<p>First we install <code class="language-plaintext highlighter-rouge">esbuild</code> through the jsbundling-rails way.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bundle exec rails js:install:esbuild
</code></pre></div></div>
<p>We will be using <a href="https://github.com/EMH333/esbuild-svelte">esbuild-svelte</a> plugin.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>yarn add esbuild-svelte
</code></pre></div></div>
<p>Add a custom build script, save the following at <code class="language-plaintext highlighter-rouge">esbuild.config.js</code> in project root directory.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#!/usr/bin/env node
var watch = process.argv.includes("--watch");
require('esbuild').build({
entryPoints: ["app/javascript/application.js"],
bundle: true,
outfile: "app/assets/builds/application.js",
plugins: [require('esbuild-svelte')()],
logLevel: "info",
watch: watch
}).catch(() => process.exit(1));
</code></pre></div></div>
<p>Replace the default esbuild build command with the build script supporting Svelte in <code class="language-plaintext highlighter-rouge">package.json</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>"scripts": {
"build:css": "tailwindcss --postcss -i ./app/assets/stylesheets/application.tailwind.css -o ./app/assets/builds/landside.css.erb",
"build": "node ./esbuild.config.js"
}
</code></pre></div></div>
<p>Alright, now we need to test if we can build Svelte components. You are free to copy and paste the following codes.</p>
<p>Save this in <code class="language-plaintext highlighter-rouge">app/javascript/svelte/DemoSvelteComponent.svelte</code></p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><script></span>
<span class="k">import</span> <span class="p">{</span>
<span class="nx">onMount</span>
<span class="p">}</span> <span class="k">from</span> <span class="dl">'</span><span class="s1">svelte</span><span class="dl">'</span><span class="p">;</span>
<span class="k">export</span> <span class="kd">let</span> <span class="nx">argName</span><span class="p">;</span>
<span class="nx">onMount</span><span class="p">(()</span> <span class="o">=></span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">argName</span> <span class="o">==</span> <span class="kc">undefined</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">argName</span> <span class="o">=</span> <span class="dl">"</span><span class="s2">General Kenobi</span><span class="dl">"</span>
<span class="p">}</span>
<span class="p">})</span>
<span class="nt"></script></span>
<span class="c"><!-- --></span>
<span class="nt"><h1</span> <span class="na">class=</span><span class="s">"italic text-blue-700"</span><span class="nt">></span>Hello there, {argName}!<span class="nt"></h1></span>
</code></pre></div></div>
<p>Save this in <code class="language-plaintext highlighter-rouge">app/javascript/application.js</code>:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="nx">DemoSvelteComponent</span> <span class="k">from</span> <span class="dl">'</span><span class="s1">./svelte/DemoSvelteComponent.svelte</span><span class="dl">'</span>
<span class="nb">window</span><span class="p">.</span><span class="nx">addEventListener</span><span class="p">(</span><span class="dl">'</span><span class="s1">load</span><span class="dl">'</span><span class="p">,</span> <span class="p">()</span> <span class="o">=></span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">t</span> <span class="o">=</span> <span class="nb">document</span><span class="p">.</span><span class="nx">querySelector</span><span class="p">(</span><span class="dl">'</span><span class="s1">[data-svelte-component="DemoSvelteComponent"]</span><span class="dl">'</span><span class="p">))</span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">app</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">DemoSvelteComponent</span><span class="p">({</span>
<span class="na">target</span><span class="p">:</span> <span class="nx">t</span>
<span class="p">});</span>
<span class="p">}</span>
<span class="p">})</span>
</code></pre></div></div>
<p>Restart your <code class="language-plaintext highlighter-rouge">bin/dev</code> and see if the message appears on the page. You should be seeing <strong>“Hello there, General Kenobi!”</strong>.</p>
<p><img src="/assets/images/rails-tailwind-svelte/step2.png" alt="Screenshot at the end of Step 2" /></p>
<h3 id="step-3-configure-tailwind-to-parse-svelte-codes">Step 3: Configure Tailwind to parse Svelte codes</h3>
<p>Inside <code class="language-plaintext highlighter-rouge">tailwind.config.js</code>, add the path to Svelte components into purge (Tailwind 2) or content attribute (Tailwind 3):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>content: [
'./app/views/**/*.html.erb',
'./app/helpers/**/*.rb',
'./app/javascript/**/*.js',
'./app/javascript/**/*.svelte'
]
</code></pre></div></div>
<p>Restart <code class="language-plaintext highlighter-rouge">bin/dev</code>, now you should be seeing the text rendered by Svelte styled by Tailwindcss.</p>
<p><img src="/assets/images/rails-tailwind-svelte/step3.png" alt="Screenshot at the end of Step 3" /></p>
<p>I would like to thank Rails core contributors for making it happen, the new workflow has been absolutely joyful to work with.</p>
<p>I hope you have enjoyed this short guide and found it useful!</p>
<p>In the next post, I will be sharing how we use Ruby on Rails as static site generator to build the new website for Talenox on <a href="https://www.talenox.com">https://www.talenox.com</a>.</p>As of the publication date of this guide, Rails 7 is about to be released and with it, comes the new cssbundling-rails and jsbundling-rails gems from the Rails core team.Malaysia Vaccine Certificate Verifier was Useless2021-07-15T00:00:00+00:002021-07-15T00:00:00+00:00https://anonoz.github.io/tech/2021/07/15/malaysia-vaccine-certificate-unsafe<p>Update: They have fixed this for iOS on 23rd July 2021, 1 week after the publication of this blog post.</p>
<p>In Malaysia, we use MySejahtera to perform contact tracing, vaccination appointments, and vaccine certification. Malaysia government has also provided another app - Vaccine Certificate Verifier - to allow the public to scan the vaccine certificate QR code in MySej app. This app will be used a lot when enough people are vaccinated and <a href="https://www.thestar.com.my/news/nation/2021/07/15/pm-consider-relaxing-movement-control-for-those-who-have-received-two-covid-19-vaccine-doses">shops reopen</a>.</p>
<p>I show you why this app is useless and harmful in its present form, as of 15th July 2021.</p>
<video src="/assets/images/vaccine-certificate/app-demo.mp4" controls=""></video>
<p>Maybe you expect the app to at least verify the website comes from MOH. It doesn’t, it’s just a simple QR code reader. The fake cert is obviously not on Ministry on Health’s server, but mine. You can try scan this:</p>
<p><img src="/assets/images/vaccine-certificate/fake-qr.png" alt="Fake vaccine certificate QR code" class="center-image" /></p>
<p>Thankfully the fix should be trivial. Just check for the hostname in the QR code, and make sure HTTPS is always used.</p>
<p>Hope you guys fix it soon before we move onto reopening!</p>
<p><em>Opengraph photo credit: Dylan Damsma</em></p>Update: They have fixed this for iOS on 23rd July 2021, 1 week after the publication of this blog post.How to sync Sublime Text packages and preferences with Git2021-01-17T00:00:00+00:002021-01-17T00:00:00+00:00https://anonoz.github.io/tech/2021/01/17/git-sync-sublime-packages-preferences<p><img src="/assets/images/sublime-text.png" alt="Sublime Text" /></p>
<p>It is possible and even pleasurable to sync Sublime Text packages and preferences across multiple workstations from different OS with Git. Here is how.</p>
<p>Outline:</p>
<ol>
<li>Find out your Sublime Text preferences directory path on Workstation A.</li>
<li>Initialise a Git repository there.</li>
<li>Push to a remote repository of your choice.</li>
<li>Find out where the preferences directory is on your destination - Workstation B.</li>
</ol>
<h3 id="pushing-it">Pushing it</h3>
<p>Tips: On Mac, just replace <code class="language-plaintext highlighter-rouge">Ctrl</code> with <code class="language-plaintext highlighter-rouge">Cmd</code> key.</p>
<ol>
<li>Run <code class="language-plaintext highlighter-rouge">Ctrl + Shift + P</code> on Windows/Linux, select <strong>Preferences: Settings</strong>.</li>
<li>Once the new is opened, <code class="language-plaintext highlighter-rouge">Ctrl + Shift + P</code> and select <strong>File: Copy Path</strong>.</li>
<li>Open a terminal, <code class="language-plaintext highlighter-rouge">cd</code> to that path excluding the ending file name.</li>
<li>Go ahead and git init, add, commit that directory.</li>
</ol>
<h3 id="pulling-it">Pulling it</h3>
<ol>
<li>Assuming you are working on a fresh installation of Sublime Text, <code class="language-plaintext highlighter-rouge">Ctrl + Shift + P</code> and select <strong>Install Package Control</strong>.</li>
<li>Repeat the steps 1-3 from above.</li>
<li>Close your Sublime Text.</li>
<li><code class="language-plaintext highlighter-rouge">git clone $gitremoteurl .</code> when you are in that <code class="language-plaintext highlighter-rouge">../Packages/User</code> directory.</li>
<li>Open your Sublime Text, now wait for a few minutes for Package Control to download all the missing packages.</li>
</ol>
<h3 id="better-way-to-do-it">Better way to do it</h3>
<p>You can install the <code class="language-plaintext highlighter-rouge">SublimeGit</code> package and just run Git inside the Preferences JSON text file. That way you won’t need to touch terminal at all.</p>How to docker-compose for Rails development: Don’t bundle install in Dockerfile2021-01-10T00:00:00+00:002021-01-10T00:00:00+00:00https://anonoz.github.io/tech/2021/01/10/rails-docker-compose-yml<p><strong>First published in 2019, updated in 2021.</strong></p>
<p>For past 12 months, we have been containerizing Rails apps for our projects, because we needed certain binaries like GnuPG 1, and Docker Swarm’s container scheduling for the high availability and scalability offered. To close the dev-prod parity, we develop our app inside Docker container using Docker Compose, and the Dockerfile we use share the same base image and system binaries as the one used for production.</p>
<p>It wasn’t easy starting out here. There are 2 problems:</p>
<ol>
<li>Not using separate Dockerfiles for development and production.</li>
<li><code class="language-plaintext highlighter-rouge">bundle install</code> inside Dockerfile for development.</li>
</ol>
<h3 id="problem-with-gems-installed-in-the-image">Problem with gems installed in the image</h3>
<p>Each instruction in a Dockerfile makes up an read-only immutable layer. For demonstration, let’s create a huge junk file, add it, and remove it in Dockerfile.</p>
<p>Create a 128MB junk file of random content in <em>terminal</em>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">dd </span><span class="k">if</span><span class="o">=</span>/dev/urandom <span class="nv">of</span><span class="o">=</span>junk <span class="nv">bs</span><span class="o">=</span>1M <span class="nv">count</span><span class="o">=</span>128
</code></pre></div></div>
<p>Add and delete the junk file in <em>Dockerfile</em>:</p>
<div class="language-docker highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">FROM</span><span class="s"> alpine</span>
<span class="k">COPY</span><span class="s"> junk .</span>
<span class="k">RUN </span><span class="nb">rm </span>junk
</code></pre></div></div>
<p>After that, build the Docker image in <em>terminal</em>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker build <span class="nt">-t</span> rm-junk <span class="nb">.</span>
docker image <span class="nb">ls</span>
</code></pre></div></div>
<p>You will get:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
rm-junk latest 7caa8295c677 6 seconds ago 139MB
</code></pre></div></div>
<p>In this example, the Dockerfile consists of 3 instructions. Since the layers built are immutable, the 3rd instruction containing <code class="language-plaintext highlighter-rouge">rm</code> command will not be able to delete the <code class="language-plaintext highlighter-rouge">junk</code> file in 2nd layer, even when you go <code class="language-plaintext highlighter-rouge">docker run -it rm-junk sh</code> and you can’t find the <code class="language-plaintext highlighter-rouge">junk</code> file inside.</p>
<p>Each layer also checks against the local cache when being built. From the sample Dockerfile above, if the <code class="language-plaintext highlighter-rouge">junk</code> file never changes, <code class="language-plaintext highlighter-rouge">RUN rm junk</code> will not be run again:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ docker build -t rm-junk .
Sending build context to Docker daemon 134.2MB
Step 1/3 : FROM alpine
---> 3f53bb00af94
Step 2/3 : COPY newfile .
---> Using cache
---> a541cb67c5be
Step 3/3 : RUN rm newfile
---> Using cache
---> c507dcbff10f
Successfully built c507dcbff10f
Successfully tagged rm-junk:latest
</code></pre></div></div>
<p>However, if there is even a tiny change in previous layers, all the subsequent layers will have the cache invalidated and will be run. That means, if you have a patch bump in <code class="language-plaintext highlighter-rouge">Gemfile.lock</code>, your entire <code class="language-plaintext highlighter-rouge">bundle install</code> and subsequent instructions will be run from scratch, costing you more space, time, and bandwidth.</p>
<p>If your project is under active development, everytime a single change is made in Gemfile/Gemfile.lock, whether adding a gem or bumping version to <a href="https://github.com/rubysec/bundler-audit">patch vulnerable gems</a>, you will have to rebuild the Docker image and run <code class="language-plaintext highlighter-rouge">bundle install</code> all over again.</p>
<p>And if you don’t clear out the old image layers often enough, you will find your machine quickly running out of disk space soon. For developers in countries with terrible Internet access, this is a serious pain.</p>
<p>Wouldn’t it be great to just download the added gem or npm package, instead of 90%+ of all other unchanged stuffs again, just like how they work outside of Docker?</p>
<h2 id="use-docker-volumes-to-store-dependencies">Use Docker volumes to store dependencies</h2>
<p>If you use Docker Compose for development, and Docker Swarm for deployment, just make sure you have 2 separate sets of Dockerfiles and docker-compose.yml. To make life simple, we use <code class="language-plaintext highlighter-rouge">Dockerfile.dev</code> and <code class="language-plaintext highlighter-rouge">docker-compose.yml</code> for development, and a <code class="language-plaintext highlighter-rouge">Dockerfile.prod</code> for production.</p>
<p>Note: I only append all Dockerfiles here with the environment for clarity’s sake. You are free to let either development or production’s Dockerfile take the vanilla, tailless <code class="language-plaintext highlighter-rouge">Dockerfile</code> name.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># docker-compose.yml</span>
<span class="na">version</span><span class="pi">:</span> <span class="s1">'</span><span class="s">3'</span>
<span class="na">services</span><span class="pi">:</span>
<span class="na">postgres</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">postgres</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="m">5432</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">postgres:/var/lib/postgresql/data</span>
<span class="na">web</span><span class="pi">:</span>
<span class="na">env_file</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">.env.dev</span>
<span class="na">build</span><span class="pi">:</span>
<span class="na">context</span><span class="pi">:</span> <span class="s">.</span>
<span class="na">dockerfile</span><span class="pi">:</span> <span class="s">Dockerfile.dev</span>
<span class="na">command</span><span class="pi">:</span> <span class="s">bundle exec rails s -p 3000 -b '0.0.0.0'</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">.:/myapp</span>
<span class="pi">-</span> <span class="s">bundler_gems:/usr/local/bundle/</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">3000:3000"</span>
<span class="na">links</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">postgres</span>
<span class="pi">-</span> <span class="s">redis</span>
<span class="na">redis</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">redis</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="m">6379</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">redis:/data</span>
<span class="na">resque</span><span class="pi">:</span>
<span class="na">build</span><span class="pi">:</span>
<span class="na">context</span><span class="pi">:</span> <span class="s">.</span>
<span class="na">dockerfile</span><span class="pi">:</span> <span class="s">Dockerfile.dev</span>
<span class="na">command</span><span class="pi">:</span> <span class="s">bundle exec rake resque:work</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">.:/myapp</span>
<span class="pi">-</span> <span class="s">bundler_gems:/usr/local/bundle/</span>
<span class="na">links</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">postgres</span>
<span class="pi">-</span> <span class="s">redis</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="na">postgres</span><span class="pi">:</span>
<span class="na">redis</span><span class="pi">:</span>
<span class="na">bundler_gems</span><span class="pi">:</span>
</code></pre></div></div>
<p>You can then have your <code class="language-plaintext highlighter-rouge">Dockerfile.dev</code> sans <code class="language-plaintext highlighter-rouge">bundle install</code>:</p>
<div class="language-docker highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Dockerfile.dev</span>
<span class="k">FROM</span><span class="s"> ruby:2.6-alpine as builder</span>
<span class="k">RUN </span>apk add <span class="nt">--no-cache</span> <span class="se">\
</span> build-base <span class="se">\
</span> busybox <span class="se">\
</span> ca-certificates <span class="se">\
</span> curl <span class="se">\
</span> git <span class="se">\
</span> gnupg1 <span class="se">\
</span> graphicsmagick <span class="se">\
</span> libffi-dev <span class="se">\
</span> libsodium-dev <span class="se">\
</span> nodejs <span class="se">\
</span> openssh-client <span class="se">\
</span> postgresql-dev <span class="se">\
</span> rsync <span class="se">\
</span> yarn
<span class="k">RUN </span><span class="nb">mkdir</span> <span class="nt">-p</span> /app
<span class="k">WORKDIR</span><span class="s"> /app</span>
<span class="k">EXPOSE</span><span class="s"> 3000</span>
<span class="k">CMD</span><span class="s"> ["bundle", "exec", "rails", "server", "-b", "0.0.0.0"]</span>
</code></pre></div></div>
<p>Now you will need to run bundle install in a separate step to get the gems:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker-compose build
docker-compose run <span class="nt">--rm</span> web bundle <span class="nb">install</span> <span class="nt">-j8</span>
docker-compose run <span class="nt">--rm</span> web bundle <span class="nb">exec </span>rails db:setup
</code></pre></div></div>
<p>After this, you can just run the 2nd command in the listing above whenever there are changes in Gemfile/Gemfile.lock.</p>
<h2 id="passing-your-ssh-key-in-as-secret-for-private-gems">Passing your SSH key in as secret for private gems</h2>
<p>Feel free to skip this section if this is irrelevant to you.</p>
<p>Your Rails app may have some private gems hosted on GitHub, Bitbucket, Gitlab. To pull those, you may either:</p>
<ol>
<li>pass in your credentials thru HTTP basic authentication,</li>
<li>authenticate with SSH keys – <strong>we will be doing this because it’s safer.</strong></li>
</ol>
<p>It is very possible to do these without leaving any secrets or credentials in your repository and your Docker images. There are plenty of ways to go about this, personally, this is how I roll:</p>
<ol>
<li>Make a <code class="language-plaintext highlighter-rouge">.secrets</code> directory, put it in <code class="language-plaintext highlighter-rouge">.gitignore</code> and <code class="language-plaintext highlighter-rouge">.dockerignore</code>.</li>
<li><code class="language-plaintext highlighter-rouge">cp ~/ssh/id_rsa .secrets/host_ssh_key</code> - we can’t just <code class="language-plaintext highlighter-rouge">ln -s</code> it in because of pesky permission issues.</li>
<li><code class="language-plaintext highlighter-rouge">chmod 600 .secrets/host_ssh_key</code> to fix permission.</li>
<li>Declare the secret in <code class="language-plaintext highlighter-rouge">docker-compose.yml</code>.</li>
<li>Add a step to symlink your secret SSH key in your <code class="language-plaintext highlighter-rouge">Dockerfile.dev</code>.</li>
</ol>
<p>Add this part in your <code class="language-plaintext highlighter-rouge">Dockerfile.dev</code>:</p>
<div class="language-docker highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">RUN </span><span class="nb">mkdir</span> ~/.ssh <span class="se">\
</span> <span class="o">&&</span> <span class="nb">ln</span> <span class="nt">-s</span> /run/secrets/host_ssh_key ~/.ssh/id_rsa <span class="se">\
</span> <span class="o">&&</span> <span class="nb">echo</span> <span class="s1">$'</span><span class="se">\
</span><span class="s1">github.com,192.30.255.112,192.30.255.113 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ== </span><span class="se">\n\
</span><span class="s1">bitbucket.org,104.192.143.1,104.192.143.2,104.192.143.3 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAubiN81eDcafrgMeLzaFPsw2kNvEcqTKl/VqLat/MaB33pZy0y3rJZtnqwR2qOOvbwKZYKiEO1O6VqNEBxKvJJelCq0dTXWT5pbO2gDXC6h6QDXCaHo6pOHGPUy+YBaGQRGuSusMEASYiWunYN0vCAI8QaXnWMXNMdFP3jHAJH0eDsoiGnLPBlBp4TNm6rYI74nMzgz3B9IikW4WVK+dc8KZJZWYjAuORU3jc1c/NPskD2ASinf8v3xnfXeukU0sJ5N6m5E8VLjObPEO+mN2t/FZTMZLiFqPWc/ALSqnMnnhwrNi2rbfg/rd/IpL8Le3pSBne8+seeFVBoGqzHM9yXw== </span><span class="se">\n\
</span><span class="s1"> '</span> <span class="o">>></span> ~/.ssh/known_hosts
</code></pre></div></div>
<p>And then amend the <code class="language-plaintext highlighter-rouge">docker-compose.yml</code>:</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gd">--- docker-compose.before.yml 2019-03-10 22:24:50.494973458 +0800
</span><span class="gi">+++ docker-compose.after.yml 2019-03-10 22:25:33.129197452 +0800
</span><span class="p">@@ -14,6 +14,8 @@</span>
context: .
dockerfile: Dockerfile.dev
command: bundle exec rails s -p 3000 -b '0.0.0.0'
<span class="gi">+ secrets:
+ - source: host_ssh_key
</span> volumes:
- .:/myapp
- bundler_gems:/usr/local/bundle/
<span class="p">@@ -40,6 +42,11 @@</span>
- postgres
- redis
<span class="gi">+secrets:
+ host_ssh_key:
+ file: ./local-secrets/host_ssh_key
+
+
</span> volumes:
postgres:
redis:
</code></pre></div></div>
<p>You should be able to bundle install the gems from private repos now.</p>
<p>If you want to pass SSH key in when building Docker image for production, read <a href="/tech/2018/05/01/rails-dockerfile.html">Dockerfile for Rails app in production and other lessons learned</a>.</p>First published in 2019, updated in 2021.Why AirAsia’s BigPay is not that safe2020-11-08T00:00:00+00:002020-11-08T00:00:00+00:00https://anonoz.github.io/security/2020/11/08/bigpay-security<p><img src="/assets/images/bigpay/bigpay-marketing.png" alt="BigPay marketing photo" /></p>
<p>In the past few months, I have received approximately 5-6 WhatsApp scam calls from different numbers, pretending to be from BigPay and Hotlink wanting to give me Covid-19 relief funds distributed by Malaysian government.</p>
<p>I can tell it is the same dude calling me using different numbers, he speaks fluent Malay. When I got the 2nd call onward, I just started trolling him by keep on replying him the wrong 6 digits SMS OTP that they have prompted BigPay to send me. The scammer is more patient than I expected, even after giving him the wrong OTP many times in a row, he did not hang up when I was trolling him.</p>
<p>This got me thinking, why is BigPay being targeted by scammers far more than any other banks out there? There are 2 reasons I am guessing right now:</p>
<ol>
<li>BigPay has only 1 factor of authentication.</li>
<li>BigPay uses only English.</li>
</ol>
<h3 id="a-primer-on-factors-of-authentication">A primer on Factors of Authentication</h3>
<p>There are 3 main factors being used right now:</p>
<ol>
<li><strong>What you know.</strong> Examples: passwords, PIN codes, your mother’s maiden name.</li>
<li><strong>What you have.</strong> Examples: handphone to receive SMS one-time password, time-based one time password like Google Authenticator, or Yubikey.</li>
<li><strong>What you are.</strong> Examples: fingerprint, face recognition, iris scans.</li>
</ol>
<p>The crux is, if somebody has your password to a system, chances are it is not hard for them to also have your PIN code. So having a login flow of asking password and PIN code in a row is not much more secure than just asking for 1 password.</p>
<p>Similarly, if someone gets hold on your credit card, chances are they can get your NRIC without much difficulty. Having a checkout page that asks for credit card numbers and NRIC together won’t make much sense.</p>
<p>However, having 2 or 3 factors of authentication together make it significantly tougher for adversaries to gain access to a system. This is why banks often ask for something only you know in your head - password, and something you have in your pocket - a phone that can receive SMS TAC, when we try to complete an outgoing transaction. It is just much much safer.</p>
<p>Unfortunately, BigPay only uses 1 factor of authentication - what you have. This makes them significantly less safe than the big banks they want to challenge against.</p>
<h3 id="the-passcode-reset-flow">The passcode reset flow</h3>
<p><img src="/assets/images/bigpay/login-flow.png" alt="Pick a country, enter phone number, key in SMS one time password" /></p>
<p>A normal login flow on a new device would be:</p>
<ol>
<li>Pick a country.</li>
<li>Enter phone number.</li>
<li>Wait for SMS OTP on the phone that you have.</li>
</ol>
<p><img src="/assets/images/bigpay/forgot-passcode.png" alt="Forgot password" /></p>
<ol>
<li>Key in the 6 digits passcode if you remember.</li>
<li>But what if you forgot it? Just key in the MyKad NRIC number and your house postcode. Both are available on your MyKad, and your cellular network provider.</li>
<li>Tada! You can now reset the passcode.</li>
</ol>
<p>Unlike other banks in Malaysia, they will first ask for password or biometrics, and they only send out SMS OTP before confirming a transaction. BigPay relies entirely on both your phone and your MyKad - something you tend to bring out together.</p>
<p>Once you logged in, the app has a tab that shows user their primary account number, expiry dates, and the 3-digits card verification codes.</p>
<p><img src="/assets/images/bigpay/card-info.png" alt="Card tab showing 16 digits PAN, expiry dates, CVC" /></p>
<h3 id="english-only">English only</h3>
<p><img src="/assets/images/bigpay/bigpay-sms.jpeg" alt="SMS with 6 digits one time code and "Don't share it with anyone, not even BigPay"" /></p>
<p>To BigPay’s credit, the one-time password SMS did say “Don’t share it with anyone, even BigPay”, so why do people still fall for it?</p>
<p>My guess is they go after the population with weak English grasp.</p>
<p>After switching my phone’s language to Malay, the app did not localise and is still in English. If someone is not proficient in English, they might just read the 6 digits blindly as instructed.</p>
<p>BigPay could do better by having us select preferred communication language when we sign up.</p>
<h3 id="do-you-trust-your-cellular-network-provider">Do you trust your cellular network provider?</h3>
<p>Twitter CEO Jack Dorsey has his Twitter account <a href="https://www.nytimes.com/2019/09/05/technology/sim-swap-jack-dorsey-hack.html">hacked with SIM-jacking</a>. It happens pretty often in the US, but not so much in Malaysia and Singapore. Again, it is a very possible attack scenario.</p>
<p>Our telcos are the weakest links to our BigPay accounts. I will not store more than RM1000 in BigPay once they are a fully fledged digital bank, if they do not incorporate 2FA/3FA in their authentication flow.</p>
<h3 id="security-advices">Security advices</h3>
<p>When I was living in Singapore back in 2018, I have my entire Maybank account emptied by the iTunes scam. Many people in <a href="https://www.channelnewsasia.com/news/singapore/apple-itunes-fraudulent-charges-banks-continue-to-assist-victims-10554658">Singapore</a> and <a href="https://www.nst.com.my/news/nation/2020/04/581238/cimb-customers-seek-clarification-direct-debit-transactions">Malaysia</a> are affected. The transactions are not done by iTunes of course, it was just the fraud syndicate using iTunes as the merchant name on statement.</p>
<p>Because I did not have a credit card, I just use my ATM debit card daily. It takes about 90 days for the bank to dispute the fraudulent transactions and refund the monies back to my saving account, if I did not have a 2nd bank account, I would not have enough money to survive 4 weeks until the next paycheck.</p>
<p>Hence, as of 2020, I still recommend you to use BigPay debit card over the usual ATM debit cards.</p>
<p>Takeaway advices:</p>
<ol>
<li>Unlink all credit and debit cards from your BigPay account, and top up only thru bank login everytime. Yes it is inconvenient, but it is very important to limit your losses to just the BigPay balance.</li>
<li>Deactivate or reduce spending limit to RM0 for all your existing bank ATM cum debit cards. Don’t get your bank account emptied.</li>
<li>Get a credit card - banks are less likely to authorize fraudulent transactions when it is their money. You do not have to pay the bank once you have submitted your dispute form.</li>
<li>Have a 2nd bank account somewhere with 1 month of living expenses.</li>
</ol>How to merge SimpleCov results on SemaphoreCI 2.02020-10-26T00:00:00+00:002020-10-26T00:00:00+00:00https://anonoz.github.io/tech/2020/10/26/simplecov-semaphoreci<h2 id="background">Background</h2>
<p>We have been using <a href="https://semaphoreci.com">SemaphoreCI</a> at <a href="https://www.talenox.com">Talenox</a> for over a year now.</p>
<p>We like Semaphore a lot because of their bare metal performance and their fair per-second pricing, which is different from other CIs charging by maximum parallelisms.</p>
<p>Having sufficient test coverage has helped us in staying clear of nasty problems along the road, I highly encourage other projects to keep it in check too!</p>
<p>Feel free to copy and paste the following scripts and YAMLs, if there are any error and problems, please contact me and let me know.</p>
<h2 id="step-1-setup-simplecov">Step 1: Setup SimpleCov</h2>
<p>Include it in your <code class="language-plaintext highlighter-rouge">Gemfile</code>, please find the latest version number and replace it:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">gem</span> <span class="s1">'simplecov'</span><span class="p">,</span> <span class="s1">'~> 0.18.0'</span><span class="p">,</span> <span class="ss">require: </span><span class="kp">false</span>
</code></pre></div></div>
<p>To make sure SimpleCov tracks the code coverage in Capybara test, paste this in your <code class="language-plaintext highlighter-rouge">bin/rails</code> before the existing code:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">#!/usr/bin/env ruby</span>
<span class="k">if</span> <span class="no">ENV</span><span class="p">[</span><span class="s1">'RAILS_ENV'</span><span class="p">]</span> <span class="o">==</span> <span class="s1">'test'</span>
<span class="nb">require</span> <span class="s1">'simplecov'</span>
<span class="no">SimpleCov</span><span class="p">.</span><span class="nf">start</span> <span class="s1">'rails'</span>
<span class="k">end</span>
<span class="c1"># ^^ Copy Above</span>
<span class="no">APP_PATH</span> <span class="o">=</span> <span class="no">File</span><span class="p">.</span><span class="nf">expand_path</span><span class="p">(</span><span class="s1">'../../config/application'</span><span class="p">,</span> <span class="kp">__FILE__</span><span class="p">)</span>
<span class="nb">require_relative</span> <span class="s1">'../config/boot'</span>
<span class="nb">require</span> <span class="s1">'rails/commands'</span>
</code></pre></div></div>
<p>In your <code class="language-plaintext highlighter-rouge">specs/spec_helper.rb</code>, paste this between all the <code class="language-plaintext highlighter-rouge">require</code> and other configuration blocks:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="no">ENV</span><span class="p">[</span><span class="s1">'CI'</span><span class="p">]</span>
<span class="no">SimpleCov</span><span class="p">.</span><span class="nf">command_name</span> <span class="s2">"</span><span class="si">#{</span><span class="no">ENV</span><span class="p">[</span><span class="s1">'SEMAPHORE_JOB_ID'</span><span class="p">]</span><span class="si">}</span><span class="s2">-</span><span class="si">#{</span><span class="no">ENV</span><span class="p">[</span><span class="s1">'TEST_ATTEMPT'</span><span class="p">]</span><span class="si">}</span><span class="s2">"</span>
<span class="k">end</span>
<span class="no">SimpleCov</span><span class="p">.</span><span class="nf">start</span> <span class="s1">'rails'</span>
</code></pre></div></div>
<h2 id="step-2-here-are-the-yaml-and-rb">Step 2: Here are the .yaml and .rb</h2>
<p>Paste this in <code class="language-plaintext highlighter-rouge">scripts/simplecov-collate.rb</code> in your project directory:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">#!/usr/bin/env ruby</span>
<span class="nb">require</span> <span class="s1">'simplecov'</span>
<span class="no">SimpleCov</span><span class="p">.</span><span class="nf">collate</span> <span class="no">Dir</span><span class="p">[</span><span class="s2">"simplecov-resultset/*/.resultset.json"</span><span class="p">]</span> <span class="k">do</span>
<span class="n">formatter</span> <span class="no">SimpleCov</span><span class="o">::</span><span class="no">Formatter</span><span class="o">::</span><span class="no">MultiFormatter</span><span class="p">.</span><span class="nf">new</span><span class="p">([</span>
<span class="no">SimpleCov</span><span class="o">::</span><span class="no">Formatter</span><span class="o">::</span><span class="no">SimpleFormatter</span><span class="p">,</span>
<span class="no">SimpleCov</span><span class="o">::</span><span class="no">Formatter</span><span class="o">::</span><span class="no">HTMLFormatter</span>
<span class="p">])</span>
<span class="k">end</span>
</code></pre></div></div>
<p>I assume:</p>
<ol>
<li>You already have SimpleCov gem included and configured in your rspec.</li>
<li>Your project directory is mounted in <code class="language-plaintext highlighter-rouge">/project-mount</code> in the Docker image that you used to test. SimpleCov somehow uses absolute path inside their result JSON files, and this is the way to make the merge successful.</li>
<li>SemaphoreCI git clones into <code class="language-plaintext highlighter-rouge">~/reponame</code> when <code class="language-plaintext highlighter-rouge">checkout</code> command is being run.</li>
</ol>
<p>Update your <code class="language-plaintext highlighter-rouge">.semaphore/semaphore.yml</code>:</p>
<ol>
<li>The <code class="language-plaintext highlighter-rouge">epilogue</code> that runs after every parallel tests, and;</li>
<li>The block that runs after ALL the tests.</li>
</ol>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># .semaphore/semaphore.yml</span>
<span class="na">blocks</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Run</span><span class="nv"> </span><span class="s">tests'</span>
<span class="na">task</span><span class="pi">:</span>
<span class="na">jobs</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s1">'</span><span class="s">E2E</span><span class="nv"> </span><span class="s">Part</span><span class="nv"> </span><span class="s">1'</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s1">'</span><span class="s">E2E</span><span class="nv"> </span><span class="s">Part</span><span class="nv"> </span><span class="s">2'</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s1">'</span><span class="s">E2E</span><span class="nv"> </span><span class="s">Part</span><span class="nv"> </span><span class="s">3'</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Models'</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s1">'</span><span class="s">API'</span>
<span class="c1"># Copy the commands in the epilogue, this runs after every different test.</span>
<span class="na">epilogue</span><span class="pi">:</span>
<span class="na">always</span><span class="pi">:</span>
<span class="na">commands</span><span class="pi">:</span>
<span class="pi">-</span> <span class="pi">|</span>
<span class="s">if [ -d coverage ]; then</span>
<span class="s">artifact push job --expire-in 3d --force coverage;</span>
<span class="s">COV_DIRNAME=simplecov-resultset/$SEMAPHORE_JOB_ID;</span>
<span class="s">sudo chown -R $USER:$USER coverage/;</span>
<span class="s">mkdir -p simplecov-resultset;</span>
<span class="s">mv coverage $COV_DIRNAME;</span>
<span class="s">artifact push workflow --expire-in 2w --force simplecov-resultset;</span>
<span class="s">fi</span>
<span class="c1"># Feel free to copy this block below!</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Check</span><span class="nv"> </span><span class="s">codebase</span><span class="nv"> </span><span class="s">quality'</span>
<span class="na">task</span><span class="pi">:</span>
<span class="na">jobs</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Test</span><span class="nv"> </span><span class="s">coverage'</span>
<span class="na">commands</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">checkout</span>
<span class="pi">-</span> <span class="pi">|</span>
<span class="s">if ! rbenv install -s; then</span>
<span class="s">git -C /home/semaphore/.rbenv/plugins/ruby-build pull &&</span>
<span class="s">rbenv install;</span>
<span class="s">fi</span>
<span class="pi">-</span> <span class="s">sudo mkdir -p /project-mount</span>
<span class="pi">-</span> <span class="s">sudo chown -R $USER:$USER /project-mount</span>
<span class="pi">-</span> <span class="s">cd ~</span>
<span class="pi">-</span> <span class="s">shopt -s dotglob; mv ~/reponame/* /project-mount</span>
<span class="pi">-</span> <span class="s">cd /project-mount</span>
<span class="pi">-</span> <span class="s">artifact pull workflow simplecov-resultset</span>
<span class="pi">-</span> <span class="s">gem install simplecov</span>
<span class="pi">-</span> <span class="s">ruby scripts/simplecov-collate.rb</span>
<span class="pi">-</span> <span class="s">cd coverage/</span>
<span class="pi">-</span> <span class="s">cd /project-mount</span>
<span class="pi">-</span> <span class="s">zip -r coverage-$SEMAPHORE_GIT_SHA coverage/</span>
<span class="pi">-</span> <span class="s">artifact push job --expire-in 2w coverage-$SEMAPHORE_GIT_SHA.zip</span>
</code></pre></div></div>
<p>Once you are done, commit and push to trigger a CI run.</p>
<h2 id="step-3-get-your-test-coverage">Step 3: Get your test coverage!</h2>
<p>If the configurations are correct and your test suites pass as usual, this is how you can find your test coverage results.</p>
<p>Go into the workflow for the latest commit, you should see there is a new block at the end after the parallel tests.</p>
<p><img src="/assets/images/simplecov-semaphore/step3-1.png" alt="Screenshot 3-1 test coverage job" class="center-image" /></p>
<p>Click <strong>Job artifacts</strong> between the console outputs and the job name. That is not the most obvious place, I know. By the time you read this, the UI may have changed again, but it should be somewhere in the page.</p>
<p><img src="/assets/images/simplecov-semaphore/step3-2.png" alt="Screenshot 3-2 job artifacts above the outputs" class="center-image" /></p>
<p>You should be able to download the zip file here.</p>
<p><img src="/assets/images/simplecov-semaphore/step3-3.png" alt="Step 3-3 Download Coverage zip." class="center-image" /></p>
<p>Download and unzip it, then open <code class="language-plaintext highlighter-rouge">index.html</code> file in your browser to view.</p>
<p>Unlike CircleCI, every artifact on SemaphoreCI cannot be viewed directly in the browser, but rather must be downloaded. It is not very convenient.</p>
<hr />
<ol>
<li>
<p><strong>Why <code class="language-plaintext highlighter-rouge">gem install simplecov</code> separately instead of just using the Docker image I have been using?</strong></p>
<p>The Docker images are usually large. Downloading just the gems needed is much quicker than pulling the image all over again. In our case, it takes almost a minute to pull the testing Docker image down, but only 20 seconds to run the whole SimpleCov merging job - from git cloning, to gem install, to uploading artifact.</p>
</li>
<li>
<p><strong>What if we did not run tests in Docker image?</strong></p>
<p>You should be able to replace <code class="language-plaintext highlighter-rouge">/project-mount</code> with the directory pah the project is git cloned into in previous test jobs.</p>
</li>
</ol>
<p>Thanks for reading. Please let me know if there are changes made to SimpleCov or SemaphoreCI that rendered this guide unusable.</p>BackgroundDocker build stuck at tzdata prompt? Could be Ubuntu 20.04 and how to fix it.2020-04-24T00:00:00+00:002020-04-24T00:00:00+00:00https://anonoz.github.io/tech/2020/04/24/docker-build-stuck-tzdata<p>Today our routine <code class="language-plaintext highlighter-rouge">docker build</code> in our CI got stuck.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Setting up tzdata (2019c-3ubuntu1) ...
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
Configuring tzdata
------------------
Please select the geographic area in which you live. Subsequent configuration
questions will narrow this down by presenting a list of cities, representing
the time zones in which they are located.
1. Africa 4. Australia 7. Atlantic 10. Pacific 13. Etc
2. America 5. Arctic 8. EurJob Finished
</code></pre></div></div>
<p>The base image we used is <code class="language-plaintext highlighter-rouge">ubuntu:latest</code>. But Ubuntu just released 20.04 Focal Fossa yesterday, and one of the changes there broke our builds.</p>
<p>Here are 2 options to go around fixing it:</p>
<h4 id="1-change-to-ubuntu1804">#1 Change to ubuntu:18.04</h4>
<p>Just change your base image down to an older stable version.</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gd">-FROM ubuntu:latest
</span><span class="gi">+FROM ubuntu:18.04
</span></code></pre></div></div>
<h4 id="2-debian_frontendnoninteractive-apt-get">#2 DEBIAN_FRONTEND=noninteractive apt-get…</h4>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gd">-RUN apt-get update && \
</span><span class="gi">+RUN apt-get update && \
+ DEBIAN_FRONTEND=noninteractive \
+ TZ=Asia/Singapore \
</span> apt-get install -y \
</code></pre></div></div>
<p>I don’t go for <code class="language-plaintext highlighter-rouge">ENV DEBIAN_FRONTEND=noninteractive</code> because that’s not an env var I wanna set permanently for the container image.</p>
<p>Anyway that’s just a quick fix, I hope they patch it soon!</p>
<p>P/S: It seems like this post is getting 200 clicks per day, if you have spare change, please consider donating to <a href="https://sharethemeal.org/donate">United Nations’ World Food Programme</a> to help people with poor food security. Pay it forward!</p>Today our routine docker build in our CI got stuck.Which MMU Comp Sci Specialization to Study in 20192019-05-20T00:00:00+00:002019-05-20T00:00:00+00:00https://anonoz.github.io/columns/2019/05/20/which-fci-specialization-to-chose<p>Hey there, I am Anonoz, class of 2017 from information system (IS) specialization. Never heard of that? Because it is gone before you studied reproduction chapter in SPM biology.</p>
<p>Choosing a specialization is extremely crucial for your academic and career success, I can’t stress enough how important it is for you to give it a fair amount of delibaration, SWOT analyses.</p>
<p>Once you have picked one and commit to it, it will become your life, you will write it on your CV, everywhere you go, you need to tell people in the industry you specialize in game development, software engineering, wrecking database (I mean IS), or data science. If you picked game development, you can only do game development, other doors are closed for you.</p>
<p>Just kidding. The truth is, whatever specialization you pick right now, does not really matter down the road. <a href="https://www.anonoz.com/degree.pdf">It won’t even show up on your degree certificate</a>.</p>
<p>But if you seriously want an answer here, it’s easy: <strong>pick the one you think you will enjoy the next few semesters the most.</strong></p>
<p>Curious about game development? Do it, even if you won’t use them ever again after graduation.</p>
<p>Want to make sick amount of money because data science is hot? You don’t necessarily have to pick data science specialization. You can either do FYP related to data science, or pick it up yourself from tons of tutorials and free courses online, concretely: <a href="https://www.coursera.org/learn/machine-learning">Coursera</a>, Udacity, <a href="http://cs231n.stanford.edu/">CS231N</a>, etc.</p>
<p>You probably have a few questions by now, or more like the questions I made up but I insist you to read them:</p>
<p><strong>1. What do I put on my resume/CV when looking for job?</strong></p>
<p>Just put “Bachelor of Computer Science from Multimedia University, Malaysia”.</p>
<p>Why? Because any specialization in early stages of career is a limiting factor. And HR outside don’t know and don’t care about what is being taught in MMU, all they know is MMU students on average are pretty OK, and you know computer science - great!</p>
<p>You can find resources and samples from your seniors on <a href="https://resumetcd.herokuapp.com">https://resumetcd.herokuapp.com</a>. Credits to Danny for maintaining it.</p>
<p><strong>2. I don’t know what I like to do, I just want to graduate, get a job, move on with my life.</strong></p>
<p>Pick the easiest specialization - SE. Study well and pass the exams on first attempt.</p>
<p>GD and DS don’t earn their reputation being tough for nothing. I barely passed visual information processing subject when it’s first being offered.</p>
<p><strong>3. If specialization is not at all important, why don’t FCI abolish it altogether and just let us pick whatever electives we like?</strong></p>
<p>I don’t know.</p>
<p>As a matter of fact, I am very against the idea of specialization, I would love to pick whatever subjects and electives to study, without being committed to it on paper.</p>
<p><strong>4. Which specialization fetches the most money?</strong></p>
<p>Singapore.</p>
<p><strong>5. Which specialization fetches the most money IN MALAYSIA?</strong></p>
<p>Chances are, they are the real specializations you can’t study in bachelor’s level.</p>
<p>You have heard about those: <a href="https://www.payscale.com/research/MY/Job=SAP_Consultant/Salary">SAP consultants</a>, <a href="https://www.payscale.com/research/MY/Skill=Red_Hat_Enterprise_Linux_%28RHEL%29/Salary">Red Hat consultants</a>. Or cross-disciplinary works like using <a href="http://iragecapital.com/data-scientist-high-frequency-trading/">data science to build high frequency trading bots</a>.</p>
<p>Go back to previous points: just pick a specialization that piques your interest and get it over with. You can study the real specialization you find interest in during or after your school.</p>
<p>You have decades of career ahead of you, you can even switch out of IT/CS if you like.</p>
<p><strong>6. Blockchain?</strong></p>
<p>It’s too nascent as a CS subfield. But if you are interested in it, I suggest you to:</p>
<ol>
<li>Take computer security elective to understand cryptography, preferrable taught by Dr. Ian Tan.</li>
<li>Read and understand <a href="https://github.com/bitcoinbook/bitcoinbook/">Mastering Bitcoin</a>.</li>
</ol>
<p>Don’t jump into coding Solidity for Ethereum dapp development yet, otherwise you are merely using a blockchain, not understanding it. Bitcoin isn’t hard to understand once you have the right fundamentals.</p>
<p>For the record, blockchain development is very active in Malaysia:</p>
<ul>
<li><a href="https://my.wobbjobs.com/users/companies/etherscan-io">Etherscan</a>, the largest Ethereum blockchain explorer is based in Shah Alam.</li>
<li><a href="https://fintechnews.my/20316/blockchain/coingecko-malaysia-transparency-data-aggregator-coinmarketcap/">CoinGecko</a> is the No. 2 coin market aggregator after coinmarketcap.</li>
</ul>
<p><strong>7. What really matters then?</strong></p>
<p>If you mean getting a job outside Malaysia:</p>
<ul>
<li>Getting your resume read.</li>
<li>Be impressive enough to get interviews.</li>
<li>Passing interviews that involve live coding, whiteboard algorithm solving, behavioural questioning.</li>
</ul>
<p>They are covered in <a href="https://resumetcd.herokuapp.com">https://resumetcd.herokuapp.com</a> bottom section.</p>
<p>If you just want to be in Malaysia:</p>
<ul>
<li>Show a bit of interest in companies.</li>
<li>Passing interviews.</li>
</ul>
<p><strong>8. What are YOU doing now, after school?</strong></p>
<p>Mostly web development, did a bit of DevOps (modern marketable term for sysadmin) to run web server cluster on AWS. Moved to Singapore, hit a snag, currently in KL again.</p>
<p>You can read up my <a href="https://www.anonoz.com/cv.pdf">resume</a> if you really curious.</p>
<p><strong>9. You didn’t answer my question!</strong></p>
<p>My bad, hit me up via <a href="mailto:hello@anonoz.com">e-mail</a>.</p>
<p><strong>10. Thank you?</strong></p>
<p>You are welcome.</p>Hey there, I am Anonoz, class of 2017 from information system (IS) specialization. Never heard of that? Because it is gone before you studied reproduction chapter in SPM biology.Watch out: Google search does not respect language HTTP headers2019-03-16T00:00:00+00:002019-03-16T00:00:00+00:00https://anonoz.github.io/tech/2019/03/16/google-search-result-wrong-language<p>I was googling ‘MDN’ for Mozilla Developer Network documentations. It showed me the Burmese site as top result.</p>
<p>But my device is set to use English, and my browsers specifically requested for contents to be shown in English, Simplified Chinese, any other Chinese, and Malay, in that particular order. I don’t know any Burmese.</p>
<p><img src="/assets/images/google-mdn-accept-lang.png" alt="Google Search Results of MDN" /></p>
<p>It just so happens that Malaysia’s ISO 3166-1 alpha-2 country code (MY) happens to be the same as Burmese ISO 639-1 language code (MY). Malay language’s ISO code is MS.</p>
<p>MDN did not do anything wrong, they chose to use <code class="language-plaintext highlighter-rouge">/$locale/*</code> for their paths, and they have set <code class="language-plaintext highlighter-rouge"><html lang="my"></code> and <code class="language-plaintext highlighter-rouge">content-language</code> response header correctly.</p>
<p>So Google chose to ignore:</p>
<ol>
<li>Client’s accept-langugage request header.</li>
<li>Server’s content-language response header.</li>
<li>HTML tag lang attribute.</li>
</ol>
<p>And simply:</p>
<ol>
<li>Determine your country from client IP address.</li>
<li>Parse the URL and just guess the first part must be a country code, it cannot possibly be language code or locale string amirite?</li>
</ol>
<p>Google is not the only offender, Facebook too:</p>
<p><img src="/assets/images/facebook-language-ignored.png" alt="Facebook knows that you don't know Malay, and renders Malay anyway." /></p>
<p>Please Google and Facebook, if my browser has explicitly told you what language I would like to be shown, respect it. If I want English, give me English. Just because I am accessing from Malaysia, does not necessarily mean I understand Malay (Saya boleh cakap bahasa tapi dia orang mungkin orang asing mah).</p>I was googling ‘MDN’ for Mozilla Developer Network documentations. It showed me the Burmese site as top result.