Import city buildings as 3D to Blender from Openstreetmap

Reading Time: 2 minutes

Buildings in 3D!

I recently figured out there’s a fantastic addon to Blender which can import buildings from Openstreetmap data. The result is that you can render a real city, or edit it to your will, with all the goodies that Blender 3D has. Neat! The possibilities are endless.

Once you learn to use the addon, it takes only about 3 minutes to make a city scenery like this:

Here’s how:

  • download the importer script (‘addon’ from now on)
  • install the addon to Blender; explicit steps below
  • the tool appears in Create > osm
  • click on ‘Select’ in the tool tab, to select an area in a new Openstreetmap web site view
  • select an area of a map you want in your Blender 3D
  • click open the ‘Export’ tab next to map
  • Click ‘Export’ button below the map
  • find your download .osm map data file
  • open Blender
  • import the .osm data: set parameters, and go!

Installation of addon (explicit steps)

  • place the .zip file in Blender’s addons folder
  • start up Blender
  • click Preferences > Addons
  • in the Addons list, activate the new addon – by ticking its box
  • now the new tool appears in Blender interface in Create > osm tab

Openstreetmap is done by people, for people. It’s basically 2D vector map, which has a lot of metadata on objects.

The key called ‘height’ in Openstreetmap data

The metadata associaeted with buildings in Openstreetmap is being used to create the 3-dimensional building. Whereas we might at first be tempted to think that Openstreetmap map data would leave us guessing the building heights, this is luckily not true. There’s a key called ‘height’ in OSM data.

You can quickly create realistic grounds for your fictional or real city. This, in fact, has been a dream of mine for a long time. I kind of vaguely imagined that there would be possibility to do the import, but how feasible? Well, it’s totally feasible!

The result to expect of city data from OSM

  • 3D lowpoly buildings
  • typically buildings consist of 6..50 faces
  • real walking paths (walks) will be represented as Blender’s “Paths”
  • you can set the vertical scale before Importing. This decides how much height a building will get for each ‘level’ in Openstreetmap
  • you can modify quite easily the .py python importer source code to change details of the import for your needs

Step-by-step: import Openstreetmap map data

Here’s the steps to follow:

  • install the addon to Blender (instructions)
  • activate the addon in Blender’s menus (User Preferences)
  • select a rectangular area from Openstreetmap web page
  • Export the area from Openstreetmap
  • Import the exported .osm file into Blender

Why?

I’m practising visualizing things and places, for a Teknovelho novel that I write. Wait.. ‘a’ Teknovelho novel? Yes.

TV 1 begins the series. Do you know h.. ahh. Okay, it’s still in Finnish. I hope that I can bring it to English one of these days!

Back to the track: it’s somewhat therapeutic to visualize things, compared to the book-writing agony.

Blender 3D – my problem axes

Reading Time: 2 minutes

You live, and you learn.

What if the global X, Y, and Z axes in Blender no longer point to orthogonal directions regarding your object, and you need to scale the object in along just one specific axis? Usually you’d do it like ‘s’ + x (for scaling along only X), but now you can’t do that! X no longer nicely points as a 90 degree normal to the object. X and your object’s faces make an arbitrary angle between 0 and 90. It’s “off”!!

That’s ugly! What now?

I ran into a curious real-world problem: when importing from real Openstreetmap vector data, Blender’s 3D axes (the directions of 3 main axis) naturally may not coincide so that things would be easy with later additions of synthetic objects.

Scaling a balcony

I had a 8-storey building, and wanted to add balconies – as important little details that would cue the viewer of the movement of elevation later on in animation. With rendered graphics, it’s important to give the human brain enough cues of motion. Otherwise the viewer feels uncertainty of the intent.

I decided: ok, a simple Cube would do for now. (Add -> Cube) This is our balcony.

But the essential thing, more important than how complex the shape would be, was to have it scaled and aligned properly to the building. Scaling by default happens in all 3 dimensions in Blender. So pressing ‘s’ when a object is selected will give you the mode of free-form scaling the object with Mouse. But since it’s a cube to begin with, I would only like to flatten it, so it would become a slab – ie scale the symmetric Cube only in depth (which happened to be approximately the X axis).

Global X doesn’t play nicely – it should be the normal to front-facing part of the building

My problem: with that much deviation between the normal of the highlighted face, and the X axis, I can’t just Scale constrained on X (‘s‘ + ‘x‘). It would result in garbage…

Programming in assembly

Reading Time: 4 minutes

Assembly is the programming language closest to a CPU. ‘Closest’ here means that the keywords in the language translate 1:1 into instructions done by the CPU. With other (higher lever) languages, there’s either an interpreter or compiler which unpacks the language and generates usually quite significantly higher number of instructions when the code is to be executed on a real CPU.

With assembly, you tell exactly the processor what you want done – at register level. Whereas higher level languages achieve a lot of operations by abstracting the ingredients into a function (subroutine) name, assembly is about the elementals of computation: manipulating bits, and registers (composed of 8, 16, 32, or 64 bits at a time) — essentially, one of the smallest storage units of digital computers.

The things we often learn as programmers is a bit more “sophisticated”, and rightly so! It’s good to work on a level we’re comfortable with.

All languages, deep down “enough”, will be compiled into assembly. For example, Java is compiled to Java bytecode, which is then run in a virtual machine called JVM. The JVM however has to eventually execute plain assembly. Same with all other languages.

A (high level) programming language can be either:

  • interpreted, or
  • compiled

The question between those two choices has mainly to do with: at which point does the conversion to machine language happen; is it “early on” (compiled languages), or during the execution (interpreted languages). Python is interpreted, while C language is a compiled language. C produces traditional executable files; whereas Python source code is run by passing the file to the Python interpreter.

Assembly is a good language to really get an understanding of what the computing hardware actually does. All modern computers are described with the van Neumann architectural model:

A computer simply can load binary values to registers; do comparison and the usual suspects like addition, subtraction, multiplication, and division; store the value back to RAM (memory); and do stuff like jump around in various parts of the code (the ‘IF… THEN’ equivalent).

At first the basic level of profiency in assembly is attained by learning the opcodes: what commands are available. In reality, even assembly commands are internally stored and executed as a sequence of microcode within the processor.

Think of registers as just 8-, 16- 32 or 64 bit variables. They are done in real gates, physical constructs in the CPU. So they “always exist”, fixed. Their value can be altered: you can load a number, or you can load a number (content) from a memory location. There are commands to

  • zero a register (make it 0)
  • add two registers (arithmetically sum)
  • subtract a register’s value from another register
  • multiply
  • divide a number in a register by another register
  • compare the values of registers (and take action: a jump = branch)

I did a lot of Intel x86 assembly programming as teen.

Is assembly really that hard?

Why does assembly have a hard-to-grasp reputation? It’s probably because of the very terse and “weird” vocabulary. Also compared to other languages, there’s so much of “nonsensical” stuff in assembly: why the heck do you “move the value 64778 to register this-or-that”.. It doesn’t seem to make any sense at all!

When you’ve learned to program in assembly, it all makes sense. But I have to admit that looking at some of the code now, in the year 2019 – that’s some 25 years later – I don’t recollect all the details anymore.

Let’s look at a image uncompression program. It’s a complete program, showing a RIX image on-screen. RIX is a format which is now almost extinct. It used to be quite popular in the wild, although very simple format. Because of being simple the .RIX was also a perfect training target for making a program that can interpret it.

.286c
KOODI SEGMENT PARA 'CODE'
ASSUME CS:KOODI, DS:TIETO
INCLUDE LIBS\JS1.INC

Set_DTA proc near
mov ah,1ah
lea dx,new_dta
int 21h
ret
Set_DTA endp

AllocMem proc near
mov ah,48h
mov bx,4096
int 21h
jnc yli1
disp_str nomem
end_prog 255
yli1:
mov alseg,ax
ret AllocMem
endp

DeAllocMem proc near
push es
mov ax,alseg
mov es,ax
mov ah,49h
int 21h
pop es
ret DeAllocMem
endp

;; Find first file, matching the search mask string defined
;; in memory area pointed to by "maski"
FindFirst proc near
mov ah,4eh
xor cx,cx
lea dx,maski
int 21h
ret
FindFirst endp

;; After we have called once the FindFirst proc,
;; continue giving next results using the same search mask string
FindNext proc near
mov ah,4fh
int 21h
ret FindNext
endp

LoadRIX proc near
lea dx,dta_name
call open_file
mov kahva,ax
mov bx,ax
call begin_file
mov bx,kahva
mov cx,64778
xor dx,dx
push ds
mov ax,alseg
mov ds,ax
call read_file
pop ds
mov bx,kahva
call close_file
ret
LoadRIX endp

SwitchPic proc near
push ds es
mov ax,0
mov w1,ax
mov w2,ax
mov ax,alseg
mov es,ax
spl:
mov cx,3
plp:
in al,60h
loop plp
mov si,w1
add si,w2
cmp si,030ah
jb eikay
cmp si,0fd09h
ja eikay
mov al,byte ptr [es:si]
push ds ax
mov ax,0a000h
mov ds,ax
pop ax
mov byte ptr [si-030ah],al
pop ds
eikay:
inc word ptr [w1]
cmp w1,0ffffh
jne yli3
pop es ds
ret
yli3:
mov ax,w1
add w2,ax
jmp spl
SwitchPic endp

ClearBuf proc near
push es
mov ax,alseg
mov es,ax
xor si,si
xor ax,ax
cbl1:
mov word ptr [es:si],ax
add si,2
cmp si,0fd10h
jb cbl1
pop es
ret ClearBuf
endp

PROSED PROC FAR
mov ax,tieto
mov ds,ax
call Set_DTA
call FindFirst
jnc yli2
disp_str norix
end_prog 255
yli2:
call AllocMem
mov ax,13h
int 10h
newpic:
call LoadRIX
push es
mov ax,alseg
mov es,ax
mov dx,000ah
xor bx,bx
mov cx,256
set_block
call SwitchPic
call FindNext
jc ulos
get_key 0
cmp al,27
je immed
jmp newpic
ulos:
get_key 0
call ClearBuf
call SwitchPic
immed:
mov ax,3
int 10h
call DeAllocMem
mov ax,0c06h
mov dh,0ffh
int 21h
end_prog 0

PROSED ENDP KOODI ENDS TIETO
SEGMENT PARA 'DATA'
w1 dw 0
w2 dw 0
maski db '*.rix',0
nomem db 'Not enough free memory (64K) to run program!$'
norix db 'No .RIX files found in current directory!$'
alseg dw 0 kahva
dw 0 new_dta db 30 dup(0) dta_name
db 13 dup(0) TIETO
ENDS
END prosed

Reliable Linux process identification, pt 2/3: Ideas

Reading Time: 3 minutes

Multicore world – does it complicate things?

Current chips, or at least the main computing units (CPUs), are made up of many cores. Some of the reasons are beyond the scope of this article. Mainly there are factors that affect the total throughput of a processor, and the thermal factors… (physical heat generation). Lowering the voltage makes thermal power problems dissipate, but, at the same time signals tend to attenuate and the very concept of having the processor precisely relay information becomes at risk.

With multicore processor you can keep some parts of the processor actually physically resting (idle / shut down) and run your necessary work load on only part of the processor.

Kernel is just… a process!

Now to blow your mind, I’ll reveal a major surprise: kernel itself is a process! I don’t want to unnecessarily make your brain wobbly, but there’s a interesting academic distinctions to be done in defining the very qualia of process.

Kernel’s role is to be a reliable “master control program”. The kernel is trusted to shepherd other processes. The hardware has to have protection rings, different levels of mode for code to be run on the CPU. Otherwise the kernel wouldn’t actually be able to protect the computing environment.

Let’s go back. Imagine a very simple computer which doesn’t have the capability of programmability. This kind of computer runs a single fixed program. Once it’s loaded, it cannot be changed. Although at first this sounds arcane, there’s lots of fixed-program embedded systems. They’re often in consumer devices, like music players, perhaps DVD/Blueray or other media players – and in a ton of other application areas more concerned with industrial control etc.

Back to the traditional Operating System like Linux or Windows.

Ok, a process is nothing but a executable (binary) file being loaded in RAM and considered as code.

The earliest experiments I did were .COM files under MS-DOS. COM files were special, simple executables, limited by a maximum size of 64 kilobytes (65536 bytes). COM programs could not dictate any specific conditions for the loader, whereas the more sophisticated .EXE files had a header area – which was used to direct the operating system about how to handle the execution.

Regardless of the steps taken, a executable image gets loaded, and is placed into such a RAM block that it is considered to be executable code. The kernel stores information about the process in its own tables.

Then starts the execution. As we’re talking about multitasking operating systems, a process gets its slice of CPU time only periodically. Often, thus, a process is not actively “on the run” 100% of the time.

If you think about computers, they’re not alive. We do however use language that implies processes are live beings. We ‘kill’ a process in Linux.

In Linux, the file format of executables is called ELF. What about those .sh and .py files and what not? They are not executables per se. They are scripts, which are loaded through an interpreter; whereas ELF files actually go into the executive “factory line” of the CPU. An instruction pointer (IP) register is pointed to the initial command of a ELF file. The ELF file may contain other “stuff” too.

Back on track: identifying processes through string-based, PID-based, or combinations thereof

A process could be identified by

  • the process name in a process listing
  • a fingerprint

PID is not a good method to identify process

Notably, the PID (a number) cannot be used to reliably identify a process. PIDs are just temporary serial numbers given by the Linux kernel to processes, as they are started. PIDs can be reused, after a process has died and thus released its PID.

What about fingerprinting processes?

Fingerprints are basically an algorithm run, with an array of bytes (content) from the process memory as the algorithm’s input.

Example: calculate hashes from a array of bytes. A hash is an algorithm well suited for fingerprinting:

  • relatively fast to compute
  • one-way
  • you can represent larger parts in short, fixed size hash values

What could we do for a process? Let’s calculate the first 512 bytes of the process image, and store <id,sum> pairs to identify processes. Each process gets a unique sum. Theoretically, for two processes to mis-identify, the chances are slim: only one in 2^512. Very astronomically small chance of error. What about reality? Depends on where (from the process) you sample the 512 bytes.

Simply the process name as identity?

Process name as identifier is pretty trivial. In Linux, the process name is the same as the “bare” executable name. Question arises, naturally: is the name of a process reliable?

Let’s try the tools in Part 3! Stay tuned.