Joy of XML with proper tools

Reading Time: 2 minutes

I was doing a recruit platform, in Java, and within there I was currently writing the spec and mechanism for format definition file for IT Skills.

Then I came upon our good old friend XML and it’s companion, the grammar definition XSD (XML Schema definition). These two have a relationship: Since XML files in general must be validly constructed (not “broken”), the validation happens by checking programmatically a XML file against the rules in a XSD file.

The XSD itself is also a language, and it’s quite flexible: you can write your own custom grammars to validate XML against. So a question arises: How is XSD defined, then?

Just enough down the rabbit hole: XSD structure

Photo by Kyle Barr on Unsplash

I’d first written just a text file, ad hoc, about some of the learning I needed to do.

The file had tags, sure, but grammar was not yet there. In order to make the writing easier, it’s useful to have autocompletion in the text editor.

Needs so far:

  • valid XML
  • schema for XML captured as a XSD
  • a validator tool for XML – takes XSD, XML -> shows results
  • editor’s autocomplete mechanism for writing compliant XML
Photo by XPS on Unsplash

Facebook Comments

Setup R for experimenting with algotrading on Windows 10 (part 1)

Reading Time: 2 minutes

Quantmod library is one of my first darlings in the exploratory experiments that I did for quantitative finance. I’d previously read about algotrading and all, but what sparked my interest really further were two things:

Photo by Priscilla Du Preez on Unsplash
  • VPRO documents on quantitative finance (Youtube)
  • especially the VPRO doc ‘The Alchemists of Wall Street
  • availability and ease of use of R and RStudio

I wanted to put my hands in the ‘savi’ (kädet saveen!) as Finns say. Pop your hood open and start meddling with things. Experimentation is the only thing that carries any digital trade forward.

Photo by SwapnIl Dwivedi on Unsplash

What is needed on Windows 10 to run R programs?

R is an interpreter, and it’s also a runtime. If you already have R installed, skip to next part!

Quick rundown of R tools

R is quite simple. Take a look at the folder and file structure:

C:\Program Files\R>dir /s /b R*.exe
C:\Program Files\R\R-4.0.2\bin\R.exe
C:\Program Files\R\R-4.0.2\bin\Rscript.exe
C:\Program Files\R\R-4.0.2\bin\i386\R.exe
C:\Program Files\R\R-4.0.2\bin\i386\Rcmd.exe
C:\Program Files\R\R-4.0.2\bin\i386\Rfe.exe
C:\Program Files\R\R-4.0.2\bin\i386\Rgui.exe
C:\Program Files\R\R-4.0.2\bin\i386\Rscript.exe
C:\Program Files\R\R-4.0.2\bin\i386\RSetReg.exe
C:\Program Files\R\R-4.0.2\bin\i386\Rterm.exe
C:\Program Files\R\R-4.0.2\bin\x64\R.exe
C:\Program Files\R\R-4.0.2\bin\x64\Rcmd.exe
C:\Program Files\R\R-4.0.2\bin\x64\Rfe.exe
C:\Program Files\R\R-4.0.2\bin\x64\Rgui.exe
C:\Program Files\R\R-4.0.2\bin\x64\Rscript.exe
C:\Program Files\R\R-4.0.2\bin\x64\RSetReg.exe
C:\Program Files\R\R-4.0.2\bin\x64\Rterm.exe

The main work horse you will be working with is ‘R.exe’

Naturally the R software installer also automatically sets up .dll libraries and does some registry keys for making sure R environment is properly set up, but you don’t usually need to do anything manually regarding these.

Photo by Todd Quackenbush on Unsplash
Facebook Comments

Boids – beautiful simulation of natural followers using an old algorithm

Reading Time: < 1 minute

Boids – a log of Learning Java EE 7 / 8

Photo by Salmen Bejaoui on Unsplash

Plan multithreading for the boid calculation

  • definition of “one round”
  • parallelism explained
  • shadow values (a copy of locations before calculation)
  • using shadow values the execution (order of picking a boid) from mesh does not matter
  • shadow buffer makes the calculations virtually as if they were done in “natural way”: all at once
  • opposite of using shadow buffer would be calculating the boids updates serially; thus the fresh-in, inter-round position of a boid would be used for all other members’ calculations; thus ordering of calculation would have an effect on the resulting positions (a externally introduced flaw, sort of order bias)
Facebook Comments

Automating software installs on Windows 10

Reading Time: 2 minutes

So, after a semi-catastrophy on my laptop, which was caused by me just jumping the gun on getting rid of ads by Acronis (backup software), I’m writing a software to ease and automate the post-bare-metal state.

Photo by Christopher Ott on Unsplash

In English, that means installation of all those software that you had on your laptop, but never actually gave them two cents of thought.

Simple idea.. loop

Think about what installation is all about? Sure. Sounds easy:

  • download the installer file
  • hash-check file in order to know integrity (safety)
  • run it
  • ta-daa!

On surface level it’s really easy. But if you stop imagining the typical installation of software, you’ll quickly remember that there are a few caveats:

  • accepting EULA or similar license dialog (keyboard/mouse)
  • choosing configuration options during installation wizard
  • confirming final options before installation starts
  • knowing when the installation wizard has finished, since Windows can’t run many installation in parallel. The previous has to end before next install can begin

You just need a file; proper user that has privilege to install new software, and a way to authenticate the file is legit and doesn’t contain malware.

So we could write a software to just programmatically drive required tools to install all the needed software. Magic. Wish all programming was this easy. Getting back to this!

Some details of getting installs right on Windows 10

Windows software installers come in a few forms:

  • .exe files
  • MSI files
  • newer MSIX – one thing I need to research a bit
  • good old .zip files

For our purposes, we are only interested in a few specific aspects:

  • is the installation file executable per se
  • if not, what are the minimal prerequisites (dependencies) needed to run file?

Does zip file cause difficulties?

Many installers come as a zip. Zip is a lempel-ziev compressed file. The repetion of consecutive identical bytes has been omitted, and there’s a few extra tricks that make the file size even smaller. If you want to know more of the theory of zip, look at wikipedia: Lempel-Ziv compression.

The problem we might encounter with zip and authenticating has to do with whether the zip can ‘cloak’ content.

This turns out not to be a problem: a hash of zip file is as good as hash of any other content – despite there’s internal structure in the file; a good hash algorithm has a property (preimage resistance) that ensures only two identically same collections of ordered bytes give out the same hash value.

I’m putting my hands down on code, and come up with the results within a couple of weeks. Hopefully! 🙂

Happy coding!

Facebook Comments

Of cleaning

Reading Time: < 1 minute

Crumbs of bread, a bit of flour. Cleaning the house is sometimes more than “just doing”. For me, it’s at its best also a great time to have a moment for thoughts on improving the design of interior.

I’m pretty visual person. Sometimes there’s immense beauty in settings like depicted here. It’s the kitchen drawer. Along a magazine I really enjoy setting my eyes on, and flipping page after page, glancing at designs, immersing myself in the feast of ideas.

Facebook Comments

Windows batch scripting basics in 2020

Reading Time: < 1 minute

Hey, imagine that you open your computer’s shell. And then you write a command. Question: What happens? And what was maybe supposed to happen?

Computers are known to be quite fragile.

More than physically wearing out or actually “breaking” a computer, you’re likely to possibly mess things up a bit. Messing happens often by:

  • typo in command (a spell error)
  • running wrong command
  • running a command that was given by someone intentionally, in an attempt to fool you
  • running a bad command – that you saw on a web page

Don’t worry. Let’s supercharge you with knowledge on these commands and the shell!

use ‘which’ in cmd

Check what your command would do, beforehand.

With shell, we can always use where command

What is a computer shell?

A shell is just another program. It’s written to take in commands, and do stuff. Shells are convenient and powerful tools. They are operated by writing commands.

Shell scripts can in addition run anything that you can write interactively, plus scripts have a few extra powerful things:

  • IF statements (allows you to make logic)
  • loop structures (run thousands of things at once, fast)
  • variables (make complex things possible)

Try the shell commands

A command looks like this:

echo Hi!

Let’s try the “echo” command in an actual Windows shell.

In Windows, first open a shell:

  • press Alt+R
  • enter ‘cmd‘ in the box
  • press enter

Now you should see a rather black and dull looking shell window. Write, using keyboard, ‘echo Hi!’, and press Enter. So just write the text e,c,h,o and space and then capital H and i followed by exclamation character (Shift+1).

The shell says back Hi!

You have used an internal shell command on Windows. See you soon!

Facebook Comments

Seth and Froggly (short story)

Reading Time: 2 minutes

Seth is 28 years. He graduated in record
time from Harvard, having just finished studies
as the great 2008 boom and bust in housing burst.
Luckily for him, his income was secured as
a Team lead in a financial portfolio and investment
company called Inkin Finance.

Froggly prototype, in a 4-port USB 2.0 HUB

He found the years following the recession somewhat
depressing at first, but thought that this
would be the perfect point to develop a more
rigorous self-improvement programme, with the
aid of software.

Inkin Finance was booming with mathematically
apt people. Creative geniuses, sometimes lacking
good “pet projects”. These guys were using their
wits all the time, for something. They couldn’t
switch their brains off. It was strange to first
come about this fact. But then again, Seth understood
he couldn’t either. He was always “on”.

With a subtle hint as a Team lead, Seth was quickly
able to get a team of two for his project – Froggly.

Froggly is a “semi-sentient AI guiding us to more
fun life”, as the team came up with a snazzy
slogan in the very first meeting.
It was 8:30PM, the team had their first casual
brainstorm in Inkin’s “fridging Kitchen” (TM) – a team-spirit
building place for many; beanball couches, perfect
coffee, hipster sentiment. And sometimes a bit of
real sunlight, even. Now there were the wading
rays of early evening.

Seth develops quickly an amicable relationship with

This personal aide is ALWAYS with him. It grows
with him. Technically they built the prototype so
that people’s smartphone browsers could be injected
with a live “Webview code” – in English, the development
can take place, while people use the software. There’s
no disruption, no updates, nothing to prevent
full steam ahead.

Froggly becomes better, and thus
the time spent doing improvements in it, becomes
more valuable. Where ever Seth goes, he finds Froggly
both useful and very entertaining. Better deals, wiser
choices, better health, more time for the One.

One day Seth asks Froggly deeper questions.
He likes to have conversations with him.
Froggly crashes. It spews a cryptic message in the
terminal. Seth’s bubble of illusion takes a slight
dent. Sure, it’s just a software. He was kind of
hoping for more.

More to come..

Facebook Comments

fraone -> upgraded to kernel 5.4.0-17

Reading Time: < 1 minute

Look at the small amount of tasks and committed RAM.

Fraone is my Frankfurt’s digitalocean Ubuntu server.

The VPS has:

  • 4 GB RAM
  • 80 GB disk

I’d previously left it running 121 days consequtively, and there was 100+ processes (‘tasks’) running. Now with a clean boot and new kernel, it’s very neat. 42 tasks (only!) and less than 256MB RAM in use. Neat!

Memory usage shows from the right end, approx 12/09 onwards that RAM usage dropped significantly. That’s the effect of a boot on a new kernel. I also shut down a service I did not have use for right now (postgresql).

Facebook Comments

Scratchpad diary: ReactJS cooking on a Digital Ocean VPS (2020)

Reading Time: 2 minutes

There’s probably a ton of ways to actually get going with developing React code. You need always anyways:

  • shell
  • Node and npm installed
  • on either Mac, Windows or Linux

One of the ways is get a VPS (a light, pay-by-use server) and do development there. I had the situation where I wanted to isolate my experiments to a rather specific server. However I wouldn’t want to spend too much on the resource, as price. A VPS was a natural choice, since I had one spare.

This Post arose as I was doing code (my “React experiments”) on a Digital Ocean Linux machine. It’s a 4GB RAM, 80GB disk VPS with a Ubuntu 18.04.2 linux version on it.

I use the server to develop on, and test run React code.

There’s a couple of things I need from the server:

  • beauty of Linux tooling – gotta hone all my knee-jerk muscle memory learning hours
  • web server (to serve the React app over web)
  • Digital Ocean’s backend IP network is way faster than at my home residential ISP (so: large gigabyte installations that sometimes happens with ‘npm’ are faster)

The last point about backhaul IP capacity difference is interesting. What it practically implies is that I am balancing between the awkwardness of latency between my laptop and Digital Ocean’s VPS (typing code), and the benefits that working remotely on the VPS gives (bandwidth for large installs).

Set up latest NodeJS package

Assuming: you’re logged on to a shell on your server

I had some leftovers from previous experiments. So first thing I did was: a) uninstall the old NodeJS, and b) install latest NodeJS (the server).

Note: if you need to uninstall previous NodeJS, type in shell:

apt purge nodejs

Then verify that removal was succesful:

node --version

and you should not have node installed. Expect the shell to throw an error message. If so, good – carry on:

Set up “nvm” for better control over Node versions

Assuming: you are still logged on in a terminal on your server

nvm is a breeze to install. It’s basically a big script. Remember to log on again (otherwise nvm settings may not take place and you’re left with the feeling that it’s broken).

nvm allows you to actually use many different node versions independent of each other, for your Node / React / javascript projects. That’s a goodie!

Photo by Taylor Vick on Unsplash
Facebook Comments