I recently figured out there’s a fantastic addon to Blender which can import buildings from Openstreetmap data. The result is that you can render a real city, or edit it to your will, with all the goodies that Blender 3D has. Neat! The possibilities are endless.
Once you learn to use the addon, it takes only about 3 minutes to make a city scenery like this:
download the importer script (‘addon’ from now on)
click on ‘Select’ in the tool tab, to select an area in a new Openstreetmap web site view
select an area of a map you want in your Blender 3D
click open the ‘Export’ tab next to map
Click ‘Export’ button below the map
find your download .osm map data file
import the .osm data: set parameters, and go!
Installation of addon (explicit steps)
place the .zip file in Blender’s addons folder
start up Blender
click Preferences > Addons
in the Addons list, activate the new addon – by ticking its box
now the new tool appears in Blender interface in Create > osm tab
Openstreetmap is done by people, for people. It’s basically 2D vector map, which has a lot of metadata on objects.
The key called ‘height’ in Openstreetmap data
The metadata associaeted with buildings in Openstreetmap is being used to create the 3-dimensional building. Whereas we might at first be tempted to think that Openstreetmap map data would leave us guessing the building heights, this is luckily not true. There’s a key called ‘height’ in OSM data.
You can quickly create realistic grounds for your fictional or real city. This, in fact, has been a dream of mine for a long time. I kind of vaguely imagined that there would be possibility to do the import, but how feasible? Well, it’s totally feasible!
The result to expect of city data from OSM
3D lowpoly buildings
typically single buildings consist of 6..50 faces
real walking paths (walks) will be represented as Blender’s “Paths”
you can set the vertical scale before Importing. This decides how much height a building will get for each ‘level’ in Openstreetmap
you can modify quite easily the .py python importer source code to change details of the import for your needs
What if the global X, Y, and Z axes in Blender no longer point to orthogonal directions regarding your object, and you need to scale the object in along just one specific axis? Usually you’d do it like ‘s’ + x (for scaling along only X), but now you can’t do that! X no longer nicely points as a 90 degree normal to the object. X and your object’s faces make an arbitrary angle between 0 and 90. It’s “off”!!
That’s ugly! What now?
I ran into a curious real-world problem: when importing from real Openstreetmap vector data, Blender’s 3D axes (the directions of 3 main axis) naturally may not coincide so that things would be easy with later additions of synthetic objects.
Scaling a balcony
I had a 8-storey building, and wanted to add balconies – as important little details that would cue the viewer of the movement of elevation later on in animation. With rendered graphics, it’s important to give the human brain enough cues of motion. Otherwise the viewer feels uncertainty of the intent.
I decided: ok, a simple Cube would do for now. (Add -> Cube) This is our balcony.
But the essential thing, more important than how complex the shape would be, was to have it scaled and aligned properly to the building. Scaling by default happens in all 3 dimensions in Blender. So pressing ‘s’ when a object is selected will give you the mode of free-form scaling the object with Mouse. But since it’s a cube to begin with, I would only like to flatten it, so it would become a slab – ie scale the symmetric Cube only in depth (which happened to be approximately the X axis).
My problem: with that much deviation between the normal of the highlighted face, and the X axis, I can’t just Scale constrained on X (‘s‘ + ‘x‘). It would result in garbage…
Assembly is the programming language closest to a CPU. ‘Closest’ here means that the keywords in the language translate 1:1 into instructions done by the CPU. With other (higher lever) languages, there’s either an interpreter or compiler which unpacks the language and generates usually quite significantly higher number of instructions when the code is to be executed on a real CPU.
With assembly, you tell exactly the processor what you want done – at register level. Whereas higher level languages achieve a lot of operations by abstracting the ingredients into a function (subroutine) name, assembly is about the elementals of computation: manipulating bits, and registers (composed of 8, 16, 32, or 64 bits at a time) — essentially, one of the smallest storage units of digital computers.
The things we often learn as programmers is a bit more “sophisticated”, and rightly so! It’s good to work on a level we’re comfortable with.
All languages, deep down “enough”, will be compiled into assembly. For example, Java is compiled to Java bytecode, which is then run in a virtual machine called JVM. The JVM however has to eventually execute plain assembly. Same with all other languages.
A (high level) programming language can be either:
The question between those two choices has mainly to do with: at which point does the conversion to machine language happen; is it “early on” (compiled languages), or during the execution (interpreted languages). Python is interpreted, while C language is a compiled language. C produces traditional executable files; whereas Python source code is run by passing the file to the Python interpreter.
Assembly is a good language to really get an understanding of what the computing hardware actually does. All modern computers are described with the van Neumann architectural model:
A computer simply can load binary values to registers; do comparison and the usual suspects like addition, subtraction, multiplication, and division; store the value back to RAM (memory); and do stuff like jump around in various parts of the code (the ‘IF… THEN’ equivalent).
At first the basic level of profiency in assembly is attained by learning the opcodes: what commands are available. In reality, even assembly commands are internally stored and executed as a sequence of microcode within the processor.
Think of registers as just 8-, 16- 32 or 64 bit variables. They are done in real gates, physical constructs in the CPU. So they “always exist”, fixed. Their value can be altered: you can load a number, or you can load a number (content) from a memory location. There are commands to
zero a register (make it 0)
add two registers (arithmetically sum)
subtract a register’s value from another register
divide a number in a register by another register
compare the values of registers (and take action: a jump = branch)
I did a lot of Intel x86 assembly programming as teen.
Is assembly really that hard?
Why does assembly have a hard-to-grasp reputation? It’s probably because of the very terse and “weird” vocabulary. Also compared to other languages, there’s so much of “nonsensical” stuff in assembly: why the heck do you “move the value 64778 to register this-or-that”.. It doesn’t seem to make any sense at all!
When you’ve learned to program in assembly, it all makes sense. But I have to admit that looking at some of the code now, in the year 2019 – that’s some 25 years later – I don’t recollect all the details anymore.
Let’s look at a image uncompression program. It’s a complete program, showing a RIX image on-screen. RIX is a format which is now almost extinct. It used to be quite popular in the wild, although very simple format. Because of being simple the .RIX was also a perfect training target for making a program that can interpret it.
.286c KOODI SEGMENT PARA 'CODE' ASSUME CS:KOODI, DS:TIETO INCLUDE LIBS\JS1.INC
Set_DTA proc near mov ah,1ah lea dx,new_dta int 21h ret Set_DTA endp
AllocMem proc near mov ah,48h mov bx,4096 int 21h jnc yli1 disp_str nomem end_prog 255 yli1: mov alseg,ax ret AllocMem endp
DeAllocMem proc near push es mov ax,alseg mov es,ax mov ah,49h int 21h pop es ret DeAllocMem endp
;; Find first file, matching the search mask string defined ;; in memory area pointed to by "maski" FindFirst proc near mov ah,4eh xor cx,cx lea dx,maski int 21h ret FindFirst endp
;; After we have called once the FindFirst proc, ;; continue giving next results using the same search mask string FindNext proc near mov ah,4fh int 21h ret FindNext endp
PROSED ENDP KOODI ENDS TIETO SEGMENT PARA 'DATA' w1 dw 0 w2 dw 0 maski db '*.rix',0 nomem db 'Not enough free memory (64K) to run program!$' norix db 'No .RIX files found in current directory!$' alseg dw 0 kahva dw 0 new_dta db 30 dup(0) dta_name db 13 dup(0) TIETO ENDS END prosed
Current chips, or at least the main computing units (CPUs), are made up of many cores. Some of the reasons are beyond the scope of this article. Mainly there are factors that affect the total throughput of a processor, and the thermal factors… (physical heat generation). Lowering the voltage makes thermal power problems dissipate, but, at the same time signals tend to attenuate and the very concept of having the processor precisely relay information becomes at risk.
With multicore processor you can keep some parts of the processor actually physically resting (idle / shut down) and run your necessary work load on only part of the processor.
Kernel is just… a process!
Now to blow your mind, I’ll reveal a major surprise: kernel itself is a process! I don’t want to unnecessarily make your brain wobbly, but there’s a interesting academic distinctions to be done in defining the very qualia of process.
Kernel’s role is to be a reliable “master control program”. The kernel is trusted to shepherd other processes. The hardware has to have protection rings, different levels of mode for code to be run on the CPU. Otherwise the kernel wouldn’t actually be able to protect the computing environment.
Let’s go back. Imagine a very simple computer which doesn’t have the capability of programmability. This kind of computer runs a single fixed program. Once it’s loaded, it cannot be changed. Although at first this sounds arcane, there’s lots of fixed-program embedded systems. They’re often in consumer devices, like music players, perhaps DVD/Blueray or other media players – and in a ton of other application areas more concerned with industrial control etc.
Back to the traditional Operating System like Linux or Windows.
Ok, a process is nothing but a executable (binary) file being loaded in RAM and considered as code.
The earliest experiments I did were .COM files under MS-DOS. COM files were special, simple executables, limited by a maximum size of 64 kilobytes (65536 bytes). COM programs could not dictate any specific conditions for the loader, whereas the more sophisticated .EXE files had a header area – which was used to direct the operating system about how to handle the execution.
Regardless of the steps taken, a executable image gets loaded, and is placed into such a RAM block that it is considered to be executable code. The kernel stores information about the process in its own tables.
Then starts the execution. As we’re talking about multitasking operating systems, a process gets its slice of CPU time only periodically. Often, thus, a process is not actively “on the run” 100% of the time.
If you think about computers, they’re not alive. We do however use language that implies processes are live beings. We ‘kill’ a process in Linux.
In Linux, the file format of executables is called ELF. What about those .sh and .py files and what not? They are not executables per se. They are scripts, which are loaded through an interpreter; whereas ELF files actually go into the executive “factory line” of the CPU. An instruction pointer (IP) register is pointed to the initial command of a ELF file. The ELF file may contain other “stuff” too.
Back on track: identifying processes through string-based, PID-based, or combinations thereof
A process could be identified by
the process name in a process listing
PID is not a good method to identify process
Notably, the PID (a number) cannot be used to reliably identify a process. PIDs are just temporary serial numbers given by the Linux kernel to processes, as they are started. PIDs can be reused, after a process has died and thus released its PID.
What about fingerprinting processes?
Fingerprints are basically an algorithm run, with an array of bytes (content) from the process memory as the algorithm’s input.
Example: calculate hashes from a array of bytes. A hash is an algorithm well suited for fingerprinting:
relatively fast to compute
you can represent larger parts in short, fixed size hash values
What could we do for a process? Let’s calculate the first 512 bytes of the process image, and store <id,sum> pairs to identify processes. Each process gets a unique sum. Theoretically, for two processes to mis-identify, the chances are slim: only one in 2^512. Very astronomically small chance of error. What about reality? Depends on where (from the process) you sample the 512 bytes.
Simply the process name as identity?
Process name as identifier is pretty trivial. In Linux, the process name is the same as the “bare” executable name. Question arises, naturally: is the name of a process reliable?
Let’s try the tools in Part 3! Stay tuned.