This web log is meant to be both technical and personal, generally detailing my exploits and adventures as well as providing notes and ideas.
- Bert JW Regeer (畢傑龍)

StackOverflow

When I was 12 I started to learn how to program, I had just together with my dad, bought my first C++ book. I don't remember the name of the book, it might have been the C++ for Dummies book, either way it came with a CD and on that CD was the DJGPP. That day I spent some time learning the new environment the book had given me, I tried writing simple programs. Just a few days after I had bought the book I wrote a very simple C++ calculator. Nothing fancy, only 32 bit integers and what you wanted to do had to be entered in a very specific way, but I felt empowered. I had learned something, I was able to do this and master the machine.

From there my grand dad introduced me to BASIC, he had written software in BASIC before and figured that would be somewhere else to start and play with, my uncle helped me set up an environment and taught me the basics of BASIC.

10 PRINT HELLO
15 PRINT WORLD
20 GOTO 10

That was the first program I ran, and I remember it mainly because I was wondering why it was scrolling by on the screen so fast. I learned backwards if you will, I started with C++ and then learned BASIC, but I really did not get started until I was in High School, 9th grade I took a programming class taught in QBASIC. While the class was fun, and exciting I wanted to always push the envelope. For one of the programs we were required to draw a house using the QBASIC built in line drawing commands. What I ended up doing was reading the co-ordinates from a file, and drawing the required shapes in the correct places. During this class the teacher knew I was flying through the assignments and he let me play around with compilers and various other programming languages. C++ was once again the programming language of choice, this time I wrote a network based chat client so that I could communicate with my friend on the other side of the room without physically having to get up and talk to him. The teacher had separated us because of too much talking and this was my solution.

From there I also started experimenting with PHP as a web programming language, I wrote various pieces of software and scripts that never took off to accomplish various things, but mostly just to see if I was able to do it, if I could accomplish the goal I had in mind. This code I still have somewhere, on a CD with various other pieces of old code, recently I took the time to go through some of my old code and I have seen how much I have grown, how much I have learned in the various years that I have been programming now. It also reminded me how much time I spent just looking for resources and information, how long it took to get answers to various different questions.

10 years later the Internet has grown, answers are now as easy as using Google with the right keywords, however even then it can take too long to find the answer to something you know someone else has come across before. This is where StackOverflow comes in, all those years that I have been learning to program I wish a site like StackOverflow had existed, it would have allowed me to easily search for answers to questions as well as ask my own, arguably stupid, questions. It is a website by programmers for programmers and is entirely based around the idea that knowledge should be shared. If you start asking Google questions about C# you will most likely find something from StackOverflow on the list of pages it gives you.

StackOverflow has taken over my life, every day I check StackOverflow for new interesting questions, check the answers and if possible I answer the question myself. Finally there is a place where programmers are able to go with all different experience backgrounds and help each other accomplish one single goal, building better software.

Updating Drupal and Cleaning Out Old Posts

I spent some time updating Drupal to the latest version that is currently available. The last install had been running on the same old version for a long time, and it was about time. Surprisingly the following update method worked very well:

Download Drupal 5.16, use the update.php script to update the blog to the 5.16 version. No errors were presented in any way shape or form which really surprised me, hats off to the Drupal developers for building a system that is stable and works correctly. After which I had some trouble, off course the old theme was now missing so logging into my Drupal was going to be an issue without an actual theme to display any of the menu's. After using some SQL-fu I regained access. Changed the theme back to the default, and things were good to go.

Next up, I grabbed Drupal 6.10. Ran its update script and everything once again went smoothly. Way smoother than any other upgrade for any of the other PHP scripts that I have had the pleasure of upgrading, then again it has been a long time since I last upgraded something as complex as Drupal. phpBB was an absolute mess to upgrade at one point in its life, have not touched it for a while so it may have improved.

Besides the newer version, there are also some modules I have now enabled. You will see that the URL's to the posts no longer contain the node/[number] format and instead are /[yyyy]/[mm]/[dd]/[post title], this should make Google and other search engines more happy than they were before, and it will make linking to a post on my website much easier. All of the old URL's still exist but will be a 301 redirect so that Google and the likes will update their search index's with the new links.

Stale content on this website, or content that was not relevant anymore has also been removed. Mostly related to things that were happening in my personal life at the time (some of the content is still up), also removing content that if it were read now, it would do more harm than good. Other content that I have removed were posts that were time/date limited and do not apply anymore to the current situation as it stands and had no real historical value.

With all of these changes in mind, I would like to apologise for those of you with feed readers that don't correctly check the date on when posts were created, you will have had all of these changes show up as yet another item in the news feed. The feed reader I personally use had no such issues since the date on the various articles had not changed, just the URL.

Sun should release ZFS under a GPL compatible license! ... Uhm, no!

Albert at Docunext is asking why ZFS is not under an GPL compatible license.


ZFS continues to get so much attention, its really bewildering to me that Sun doesn’t release it under a GPL compatible license.

- ZFS?!!?!?!?

I posted a comment over at his blog, but I also wanted to post the same comment here. I have never really appreciated the whole Linux mindset of take take take, and wanted to have my position be known on this here blog because I feel it is an important matter that people ask me each time I mention ZFS to them.

--

Why should Sun be required to release their file system under a GPL compatible license just so that the GNU tards can take it and make it their own? Look at what happened to the wireless drivers that OpenBSD created, they were re-licensed under GPL meaning any new changes could not be brought back into the OpenBSD versions. The people doing so thought it was perfectly fine and so did many of the other Linux users. Yet when an OpenBSD developer had pieces of Linux code in his code so that he could slowly write his own driver for a certain piece of hardware there were harsh cries being made. Take take take, what do you give back?

Linux is not the end all be all open source operating system, I'll be honest, I have not used Linux for any of my contract work in the last year. Everything is FreeBSD. Why? Because the OS is more stable, everything comes as a whole, not everything is yet a different package from a random source. The kernel is generally secure, and so are the userland applications. Major bugs are less frequent and in general the whole thing is more solid because of the way it is put together.

Also, why can't Linus and the Linux kernel adapt and allow other licenses to sit in the kernel along side the rest of the kernel? This seems to be working for Mac OS X, and for FreeBSD. Where you can even choose to include ext2 support, thereby "tainting" your kernel with GPL code. Why can't Linux adapt the same possibility? Seems unfair that Sun should release their file system under a license that would benefit no-one else but the Linux community! What do they end up getting back for it?

Same thing is happening with DTrace, the Linux guys are re-making it from scratch because they like the feature set but they don't like the license. Once again, Mac OS X and FreeBSD seem to have no issues.

All the mainline kernel developers have been doing is putting ZFS down, Andrew Morton for example called it a Rampant Layering Violation (http://lkml.org/lkml/2006/6/9/389) (Jeff Bonwick replies: http://blogs.sun.com/bonwick/entry/rampant_layering_violation). Yet, now btrfs is supposed to make everything better, yet it has those exact same layering violations. What gives?

Sun should and probably will never release ZFS under a GPL or GPL compatible license. It just does not make sense. There is no business need for them to do so, nor is there any return for them when they do. The people that are serious about storage right now are able to use ZFS through Solaris 10, OpenSolaris, FreeBSD, and Mac OS X Server.

Want to own a limited edition geeky watch?

My room mate Kris has just gone live with his website and he is selling his custom built watch: Multiwatch. It is a watch that is able to display time in a variety of different means, for example binary, hexidecimal, octal and decimal. The entire watch is designed using only one button, and is activated by tapping the front of the watch to turn it on. With yet another tap you are able to change the time format from one of the above mentioned time formats to another format.

There are many different pictures of the watch available online at the website: MultiWatch pictures, check them out. Killbot 9000 is even wearing one!

As each watch is hand assembled they are all numbered, and thus you are guaranteed a unique watch that no-one else in the world will have. Once again, it is a limited edition watch made for the geek within you. There are two hundred being sold total. Once that magical number is reached there are no more, get your order in now!

Multiwatch product page at KBEmbedded

NFS Read/Write errors?

Double check that your userid's match on all systems.

Hosting NFS shares from a Zone

I am currently trying to get my web development environment all set up. While I like using vi, I really do prefer TextMate on my Mac OS X to do the real heavy lifting and all of the editing.

As I was working on getting my homedir on my zone to be shared over NFS I was getting errors that NFS was not a valid protocol from sharemgr, the program that is used to add new shares to the system.

xistence@webdev.network.lan:~# sharemgr create -P nfs homedir  
Invalid protocol specified: nfs

A little bit of Googling brought me to a mailling list message saying that some server packages may be missing. This is entirely possible since my Zone was created with a minimal set of packages, so I log into the global host where NFS sharing does work and run the following:

xistence@Keyhole.network.lan:~# pkg list | grep nfs
SUNWnfsc                                      0.5.11-0.101    installed  ----
SUNWnfsckr                                    0.5.11-0.101    installed  ----
SUNWnfss                                      0.5.11-0.101    installed  ----
SUNWnfsskr                                    0.5.11-0.101    installed  ----

Ah, so those are probably not installed in the zone, quick pkg install later:

xistence@webdev.network.lan:~# pkg install SUNWnfsc SUNWnfsckr SUNWnfss SUNWnfsskr
DOWNLOAD                                    PKGS       FILES     XFER (MB)
Completed                                    3/3       17/17     0.28/0.28 

PHASE                                        ACTIONS
Install Phase                                  55/55 
PHASE                                          ITEMS
Reading Existing Index                           9/9 
Indexing Packages                                3/3

Now, lets re-run the sharemgr command again and see if this time we do indeed get our group created:

xistence@webdev.network.lan:~# sharemgr create -P nfs homedir
xistence@webdev.network.lan:~# sharemgr show -vp
default nfs=()
zfs
homedir nfs=()

Yes, there we go. Now I can add a share to that group. Until more errors start showing up:

xistence@webdev.network.lan:~# sharemgr add-share -s /export/home/xistence/ homedir
NFS: Cannot share remote filesystem: /export/home/xistence
NFS: Service needs to be enabled by a privileged user
Failed to enable share for "nfs": system error

This time Google does not help out, instead it gives me the option to search without quotes around the search terms which gives me absolutely nothing.

Well, lets see if the services are running in the first place:

xistence@webdev.network.lan:~# svcs -a | grep nfs    
disabled       22:19:04 svc:/network/nfs/client:default
disabled       22:24:26 svc:/network/nfs/server:default
online         22:24:20 svc:/network/nfs/rquota:default
online         22:24:20 svc:/network/nfs/status:default
online         22:24:20 svc:/network/nfs/mapid:default
online         22:24:20 svc:/network/nfs/cbd:default
online         22:24:21 svc:/network/nfs/nlockmgr:default

Nope, for some reason the nfs server keeps being disabled eventhough I did enable it using svcadmin.

Okay, lets take a look with svcs what is going on:

xistence@webdev.network.lan:/# svcs -l nfs/server
fmri         svc:/network/nfs/server:default
name         NFS server
enabled      false (temporary)
state        disabled
next_state   none
state_time   Fri Feb 20 22:25:26 2009
logfile      /var/svc/log/network-nfs-server:default.log
restarter    svc:/system/svc/restarter:default
contract_id  
dependency   require_any/error svc:/milestone/network (online)
dependency   require_all/error svc:/network/nfs/nlockmgr (online)
dependency   optional_all/error svc:/network/nfs/mapid (online)
dependency   require_all/restart svc:/network/rpc/bind (online)
dependency   optional_all/none svc:/network/rpc/keyserv (disabled)
dependency   optional_all/none svc:/network/rpc/gss (absent)
dependency   optional_all/none svc:/network/shares/group (multiple)
dependency   require_all/error svc:/system/filesystem/local (online)

Ah, it has a log file, lets see what that log file says:

xistence@webdev.network.lan:/# cat /var/svc/log/network-nfs-server\:default.log 
[ Feb 20 22:11:07 Disabled. ]
[ Feb 20 22:11:07 Rereading configuration. ]
[ Feb 20 22:18:43 Enabled. ]
[ Feb 20 22:19:04 Enabled. ]
[ Feb 20 22:24:21 Executing start method ("/lib/svc/method/nfs-server start"). ]
The NFS server is not supported in a local zone

Eh? What? That seems weird, but it is off course plausible. Zones are new to Solaris and thus not everything may be virtualised yet. Taking that query to google brings me to a web page asking the same question I am asking: Local zone as NFS server. Taking a look it brings me to a bug report and ultimately to: Bug ID: 4964859 RFE: Zones should be able to be NFS servers. Well, I guess that ends that little stunt. I will be sharing it from the global zone. While this is acceptable to me, I do not believe that it is acceptable to most other people using Zones, and while it is unfortunate but it does not look like there is a whole lot of activity on the bug, as apparently there are no resources available.

Well, I learned something new, and wasted a few minutes of my time attempting to set this up. While generally I don't post my failures I find it important as other people may run into the same errors and have the same issue with getting things up and running.

I ended up doing the following:

sharemgr create -P nfs webdev-homedir

Then I followed that up with some servers that should be allowed access, and others that should not:

sharemgr set -P nfs -S sys -p 'rw=10.10.10.226 none=*' webdev-homedir

And then finally I added the share I wanted to share all along:

sharemgr add-share -r "homedir" -d "xistence home dir" -s /storage/zones/dev-web/root/export/home/xistence webdev-homedir

This creates a long NFS share to type in, since the full path now becomes:

nfs://keyhole/storage/zones/dev-web/root/export/home/xistence

Which is not that big of a problem for me, especially since I just save it in my favourites in Mac OS X and everything is peachy. It works like a charm.

FIRST Robotics mentoring

The last six weeks have been the fastest few weeks of this new semester that I am currently in. FIRST Robotics season had started again and I tagged along with two friends of mine, Zach and Jason, to mentor high school students at Hamilton High School in Chandler, AZ. The team number for those of you interested in 698, which is not that far away from 637 the team that I was on in 2006.

It was interesting being back in a High School. First and foremost because I had not realised how much I had grown up in just two years of University, and how different everything was. I forgot all about the scheduled classes with bells to signify when to go the next class, the weird schedules whereby classes shift around depending on the day, and various other oddities (the chairs for example are a lot smaller, and have desks attached to them). When I say I had not realised how much I had grown up, I mean that in the worst possible way. While me and my friends are able to joke around when we need to get something done we can sit down and get it done. Most of these students would sit around and joke and would not be able to get anything done no matter how long you gave them, they needed serious directions and to be told exactly what to do. Micromanaging in this environment is an absolute must. As for the differences, I am in University that means I get to decide if I got to class or not, I get to decide what classes I take, and I have to keep track of my schedule because my classes are not going to be offered one after the other, and there is not a single bell or paging system in the building.

Team 698 was off to a rocky start this build season, for various reasons, but the main issue was that the leader of the team was not leading but instead was trying to build the entire robot by himself, thereby killing team morale, and their reasons for even working, as everything they'd come up with would not be good enough. Once this was rectified and the team's mentors including myself started to pull the team in the right direction everything started rolling.

Tutoring people in high school is actually an awesome experience, and while I generally hate teaching people (Google is your best friend) I found that I enjoyed working with the students. Even if they could be a little childish, easy to derail and get off track from the task at hand. That being said, I have a much greater respect for my High School teachers now that I have gone through what they have to go through every single day. That is no easy task and it takes a lot of energy. The students in Robotics are probably a different breed though and they were more eager to learn and build new ideas than probably most other high school students.

The end result is that the robot was shipped. There are still some things that need to be worked on, but for now there is a small break before the Arizona FIRST Robotics Regionals. If you get the chance to be a mentor on a FIRST Robotics team I would totally suggest it. It is absolutely fantastic and more people should get involved. When I was a student and on the FIRST Robotics team it was an absolutely exhilarating experience, and something that even to this day I still remember vividly. I may have had some major arguments with people on the team (most notably my Physics teacher at the time), but we worked through them, and it is to this day that I remember something my Physics teacher, Mr. Giraud, said to me:

"You can write software expecting the hardware to be perfect, unfortunately hardware is not perfect and you have to fix it in code." (May not be exact, but the basic gist is there)

That has stayed with me, and it is something I will probably always remember. It has certainly changed the way I program for embedded devices.

So you want to upgrade your zone

Recently I was working on my OpenSolaris machine (file server, ZFS rocks, will write more on that later), and I have one non-global zone, one that use for web development, which is aptly called dev-web.

So the following shows up when I run zoneadm:

xistence@Keyhole.network.lan:~# zoneadm list -iv
  ID NAME             STATUS     PATH                           BRAND    IP    
   0 global           running    /                              native   shared
   4 dev-web          running    /storage/zones/dev-web         ipkg     shared

The thing is, I had upgraded the global zone with the latest update available for the version (snv_101b):

pkg image-update -v

This had not upgraded my one none-global zone. And running pkg image-update from within the zone itself is not possible, because you can't do an upgrade on a "live" system, mainly because the image-update wants to create a new bootable environment, something that was already created when I upgraded the global zone. So what we have to do is mount the non-global zone to /mnt and tell pkg with -R where to find it and upgrade it anyway!

First we are going to halt the current zone, since I am mean and nothing important is running on the test bed system, I just used:

zoneadm -z dev-web halt

However, the better way is to off course use the shutdown command:

zlogin dev-web shutdown

and then for good measure a halt!

Next up, looking at what we are going to be mounting.

zfs list
NAME                               USED  AVAIL  REFER  MOUNTPOINT
rpool                             11.9G  42.7G    75K  /rpool
rpool/ROOT                        2.93G  42.7G    18K  legacy
rpool/ROOT/opensolaris            7.34M  42.7G  2.74G  /
rpool/ROOT/opensolaris-1          2.92G  42.7G  2.74G  /
rpool/dump                        1.87G  42.7G  1.87G  -
rpool/export                      5.27G  42.7G    19K  /export
rpool/export/home                 5.27G  42.7G    21K  /export/home
rpool/export/home/guest           5.24G  14.8G  5.24G  /export/home/guest
rpool/export/home/xistence        37.0M  42.7G  37.0M  /export/home/xistence
rpool/swap                        1.87G  43.0G  1.51G  -
storage                            493G  3.08T  35.1K  /storage
storage/media                      398G  3.08T   398G  /storage/media
storage/virtualbox                2.10G  3.08T  2.10G  /storage/virtualbox
storage/xistence                  91.7G  3.08T  91.7G  /storage/xistence
storage/zones                      957M  99.1G  30.4K  /storage/zones
storage/zones/dev-web              957M  19.1G  32.0K  /storage/zones/dev-web
storage/zones/dev-web/ROOT         957M  19.1G  28.8K  legacy
storage/zones/dev-web/ROOT/zbe    1.60M  19.1G   936M  legacy
storage/zones/dev-web/ROOT/zbe-1   955M  19.1G   935M  legacy

When pkg image-update was run on the global zone it created a new bootable environment named opensolaris-1, the cool thing is, that beadm at the same time will also create a new bootable environment for your zones. That way you can upgrade your zones afterwards, and if stuff does not work, you can revert the ENTIRE machine back to the previous state (ZFS is cool like that), thereby also making sure that your zones are reverted so that there are no incompatibilities.

So what we are looking for in this case is a zbe-1, this is the new root for the zone that we need to update, so we now need to mount it.

mount -F zfs storage/zones/dev-web/ROOT/zbe-1 /mnt

Note, that there is no / in front of storage, this is because we are specifying a pool name, since there is no "real" path that is defined as /storage/zones/dev-web/ROOT/zbe-1. Now that it is mounted, we are able to pass the -R flag to pkg, to get it to update our zone:

xistence@Keyhole.network.lan:~# pkg -R /mnt image-update -v
Creating Plan / Before evaluation:     
UNEVALUATED:
+pkg:/entire@0.5.11,5.11-0.101:20081204T010954Z

After evaluation:
pkg:/entire@0.5.11,5.11-0.101:20081119T235706Z -> pkg:/entire@0.5.11,5.11-0.101:20081204T010954Z
Actuators:

None
PHASE                                        ACTIONS
Update Phase                                     1/1 
PHASE                                          ITEMS
Reading Existing Index                           9/9 
Indexing Packages                                1/1 

---------------------------------------------------------------------------
NOTE: Please review release notes posted at:
   http://opensolaris.org/os/project/indiana/resources/relnotes/200811/x86/
---------------------------------------------------------------------------

xistence@Keyhole.network.lan:~# umount /mnt

Voilá, and the deed is done. The last command is to off course unmount the zone, that we can then issue a zoneadm boot command to start it back up:

zoneadm -z dev-web boot

And then on the zone after we log into it (over SSH in my case):

xistence@webdev.network.lan:~# pkg list -u
No installed packages have available updates

Which is exactly what we wanted!

Rogue DCHP servers -- Malware becomes more sophisticated

It has been a while since an idea that I have been floating around in my head has come true in the real world. Ever since I experimented with ettercap almost 2 years ago, I was wondering how long until we would see the idea of being able to beat a DHCP server in a race condition would be implemented on some wider scale to do phishing attacks on entire ISP's.

Luckily it is not as bad yet, however according to SANS Internet Storm Center there is a new DNS changing piece of Malware that installs a valid TCP/IP driver in Windows to have raw packet access, sets up a listener and emulates a DHCP server. Whenever it sees a DHCP request it replies with its own DHCP reply, hopefully before the real DHCP server gets a chance to do so, and sets the DNS resolver IP's to ones located in the Ukraine.

Very interesting, how secure are ISP's against these type of attacks? Could I set up a fake DHCP server on my outbound connection and reply to DHCP packets? Food for thought.

Rogue DHCP servers article at SANS

Using DJB's daemontools and netcat to bounce an incoming request around the world

Lately I have been moving the data from one server to another server, located halfway across the globe, and I needed some way to forward all incoming requests from the old server to the new server. This had to be done so that DNS could take its time in updating while everything was now already being served from the new location.

What I ended up doing was using daemontools along with netcat to pipe the request around the world. Here are the steps I took in doing so:

mkdir /usr/local/redirect/
cd /usr/local/redirect/
mkdir smtp smtp/env
cd smtp

Put the following in a file named run:

#!/bin/sh
exec 2>&1 \
envdir ./env \
sh -c '
    case "$REMOTENAME" in h) H=;; p) H=p;; *) H=H;; esac
    case "$REMOTEINFO" in r) R=;; [0-9]*) R="t$REMOTEINFO";; *) R=R;; esac
    exec \
    /usr/local/bin/tcpserver \
        -vDU"$H$R" \
        ${LOCALNAME+"-l$LOCALNAME"} \
        ${BACKLOG+"-b$BACKLOG"} \
        ${CONCURRENCY+"-c$CONCURRENCY"} \
        -- "${IP-0}" "${PORT}" \
        /usr/bin/nc "${REMOTEHOST}" "${REMOTEPORT}"
'

Make it executable:

chmod +x run

Then we need to set up a few environment variables:

cd env
echo "example.net" > REMOTEHOST
echo "25" > REMOTEPORT
echo "25" > PORT
echo `hostname` > LOCALNAME
echo "200" > CONCURRENCY

I made REMOTEPORT and PORT be separate on purpose, in one case I now had more IP's than before, so instead of having SSL running on a separate port it was running on the default port, and I needed a clean way to forward that.

Now just add it to your services folder as a symlink and it will automatically be started, from there it will do it's job! I also suggest adding some simple logging, or discarding all off the output from tcpserver.

Syndicate content