Recap on setting up Raspberry Pi emulation on MacOSX

The purpose is not completely guide how to setup Raspberry Pi emulation on MacOSX but rather to note my practice and with specific problem solving on my own environment

This is really interesting topic, it recalled my old days with Linux programming, configure and build Linux kernel, but I have to give up my time, just want to get the Raspberry emulation up and run, that's it!

I had referenced many sources (see below) in order to setup and solve problems and when I finished my setup, I got the conclusion:

1. Install brew (http://brew.sh/) on your MacOSX first
2. Use brew to install common tools and libraries
3. Download and build ARM build tools chain, this will be used to compile Raspberry Kernel
4. Download, configure and build Raspberry Kernel using tools chain above
5. Install QEMU and run Raspberry emulation

I would also make a copy of sources in put it here, it's just to make sure the guide still here and convenient for you to follow, or in the worst case if the source goes down (that I faced sometime when reading guide that has referenced sources)

• In order to install brew, it's quite simple so I would like to skip this step and move on from the step #2 install common tools and libraries, #3 download and build ARM tools chain

brew install mpfr gmp libmpc libelf texinfo

mkdir ~/rpi

mkdir ~/rpi/arm-cs-tools

git clone https://github.com/jsnyder/arm-eabi-toolchain.git

cd arm-eabi-toolchain

PREFIX=$HOME/rpi/arm-cs-tools make install-cross

make clean

echo "export PATH=$HOME/rpi/arm-cs-tools/bin:$PATH" >> ~/.zshrc

*Please be noticed here: If you use the other shell rather than zsh you have to change to output of export accordingly, in my case, it's bash shell - .bash_profile. It also depends on your purpose if you want to setup tools chain for your own usage only or for global system you might want to output export command to appropriate shell's resource or profile file

You also have to update XCode in your MacOSX to latest version, the reason is the build tools, and libraries, dependencies might depend on XCode's libraries in order to build your 'build tools' or Kernel

• Clone Raspberry Kernel:

mkdir ~/rpi/kernel

cd ~/rpi/kernel

git clone https://github.com/raspberrypi/linux.git

cd linux

• Make a copy of predefined Kernel Configuration, and modify it as your wishes

cp arch/arm/configs/bcmrpi_cutdown_defconfig .config

make ARCH=arm CROSS_COMPILE=~/rpi/arm-cs-tools/bin/arm-none-eabi- menuconfig

*The .config file is where you can modify your Kernel configuration, and the build process will follow up defined configuration in there

In my case, there is no bcmrpi_cutdown_defconfig in the folder, so I have to google, download and put it in configs folder

At this point, I leave all the default (by pressing ENTER) to the questions from command in order to build .config file

If you don't really know what changes (configuration files, aka: .config file) written to your Kernel configuration you can issue command below to see what files had changed last 10 minutes

find ~/rpi/kernel/arch/arm/configs -mmin -10

• Make sure SYMBOLS are defined

During compilation process it will appear several warnings that you can ignore but not for the errors. There are many copies of the orginial tutorial on this topic (not sure which one is the original) but none of them correct this error of missing SYMBOLS, except comment on this link: http://www.mluis.com/post/27090213894/rpi-osx. Thanks to Alan for his discover https://disqus.com/by/alanjjenkins/ atleast it's applicable for my environment where it missed #define R_X86_64_64 0 and #define R_X86_64_NONE 0.

Also, the libelf.h is in libelf folder, in my case, so I have to modify a bit the include to #include

sudo touch /usr/local/include/elf.h

#include

#define R_386_NONE 0

#define R_386_32 1

#define R_386_PC32 2

#define R_ARM_NONE 0

#define R_ARM_PC24 1

#define R_ARM_ABS32 2

#define R_MIPS_NONE 0

#define R_MIPS_16 1

#define R_MIPS_32 2

#define R_MIPS_REL32 3

#define R_MIPS_26 4

#define R_MIPS_HI16 5

#define R_MIPS_LO16 6

#define R_X86_64_64 0

#define R_X86_64_NONE 0

• Build your Kernel

make ARCH=arm CROSS_COMPILE=~/rpi/arm-cs-tools/bin/arm-none-eabi- -k

During Kernel compilation process, you might see errors alike below and the process will be stopped (thanks to http://blog.icanit.ru/2014/05/dev-c-kernel-rpi-os-x.html)

UNROLL lib/raid6/int1.c
 awk: can't open file -vN=1
  source line number 1 source file -vN=1
  context is
       >>> <<<  
  CC [M] lib/raid6/int1.o
 arm-none-eabi-gcc: ошибка: lib/raid6/int1.c: No such file or directory

It happened the same in my environment so I just follow the steps guided in the link to fix it. It is not really exactly the line number mentioned in the link. But you still can compare the text below with relevant text in your file

Fix editing lib / raid6 / Makefile file line 14.15 (Adds problems after -v, -N and change the order in the OS X awk sensitive):

    cmd_unroll = $(AWK) -v N=$(UNROLL) -f $(srctree)/$(src)/unroll.awk \  
           < $< > $@ || ( rm -f $@ && exit 1 )  

The same:

Building modules, stage 2.  
 xargs: illegal option -- r  




To fix it, we need to get rid of the use of the xargs -r flag, which yne supported by BSD version of this tool 
should be corrected in line 63 scripts / Makefile.modpost file at:

 MODLISTCMD := find $(MODVERDIR) -name '*.mod' | grep . | xargs grep -h '\.ko$$' | sort -u  




*Download and build Raspberry Kernel, this is the most painful part for me or anyone who has no idea or forgot the way to build Linux Kernel. It involves not only download and build but also difficult steps to configure your Kernel before building, and also configure, download dependencies, libraries that the build process relies on.

This step is also really important since how your Raspberry works later on depends on this step, if your Raspberry supports or not features, eg: kinda of filesystem, network, etc.. In this step, it even says successful but it's not guaranteed that you can use your Kernel to boot up your Raspberry. The boot up process will actually check configure, hardware, etc.. So if you mis-configure any parts of it aka: drivers, processor, etc. It will be fail during boot up, and you have to analyse the errors, go back and reconfigure your Kernel

It's also recommended to turn on DEBUG mode in your Kernel configuration file so the boot up process later will show you more details of the errors. Painful & Good luck!

• Download & Install QEMU

I follow this link http://raspberrypi.stackexchange.com/questions/7081/how-to-successfully-emulate-rpi-on-osx/15891 to install QEMU

It has very clear explanation why you need to install apple-gcc42: "Due to a bug of a white screen hanging QEMU if compiled with llvm one must install the package apple-gcc42 from the homebrew’s dupes repository."

brew install homebrew/dupes/apple-gcc42

Then compile QEMU

brew install qemu —use-gcc

At this point, you reach mostly the end of the procedure, one more step before running QEMU is to make sure your brew got upgraded, updated to latest

$ brew update

$ brew upgrade

In my case, I got errors complaining

dyld: Library not loaded: /usr/local/lib/libpng15.15.dylib
Referenced from: /usr/local/bin/php
Reason: image not found

See more: http://stackoverflow.com/questions/22631026/dyld-library-not-loaded-usr-local-lib-libpng16-16-dylib-with-anything-php-rel

• Launch QEMU with Kernel Image, and Harddisk Image

Bravo - you have done the most important steps, you now can enjoy your hard work already

qemu-system-arm -M versatilepb -cpu arm1176 -kernel kernel-qemu-3.10.25-wheezy -hda 2012-07-15-wheezy-raspian-minimal.img -append "root=/dev/sda2"

*Please be noticed, if you copy command above and paste it into your console you might have a problem with this character " and it will failed your command

The parameter "root=/dev/sda2" pointing root filesystem to your image (harddisk), in this case it's sda2 (the partition #2 on your sda). For some instances, it might fail to load the root filesystem since the Kernel could not recognize the filesystem and you have to point root to other partition, eg: sda1, sda3, it depends on your disk image

Not syncing: No working init found. Try passing init= option to kernel. See Linux Documentatio/init.txt fot guidance.

You might want to check out this page: https://github.com/torvalds/linux/blob/master/Documentation/init.txt to understand more how init process works in Linux

There is another reason is your disk image is corrupted so that the boot up process couldn't recognize it, you have to use other computer to mount that disk image and fix it. This link will give you the detail explanation http://raspberrypi.stackexchange.com/questions/41965/error-on-boot-no-working-init-found. In sort the command to check and fix your disk image, if it's ext2 filesystem

e2fsck

• Get your Raspberry up to date

*Before doing update/upgrade your Raspberry emulation, please make sure your QEMU running on MacOSX with internet access. For some cases, your disk image isn't configure for internet access, you might want to see below for how-to

sudo apt-get update
sudo apt-get upgrade

References:

• Main source: https://thecustomizewindows.com/2015/03/raspberry-pi-virtual-appliance-emulation-on-os-x/
• Problem Solving: Inline links
• Prebuilt Kernel Image: https://github.com/dhruvvyas90/qemu-rpi-kernel (there are many links on internet but it all was died, for that reason dhruvvyas90 had made this source available for everyone)
• Prebuilt Hard Disk Image: The ariticle: http://www.cnx-software.com/2012/07/31/84-mb-minimal-raspbian-armhf-image-for-raspberry-pi/ & The image: https://dl.dropbox.com/u/45842273/2012-07-15-wheezy-raspian-minimal.img.7z
• The official Raspbian page: https://www.raspberrypi.org/downloads/
• Learn about QEMU command: http://www.linuxcertif.com/man/1/qemu-system-arm/

Useful commands (and rarely used as well) during working with Debian packaging

I am working on deployment of our demo server, I am using puppet for most of deployment except this one, using Debian package instead... um... I don't really like it (and finally I found it dumb). I talked to myself, at least I can learn something new and here it is.

The post doesn't try to make clear on the use of these commands, it's more about notes for my future reference only... sorry!

1/ How to get change log version? (my dpkg-parsechangelog version 1.15.11)
Command: dpkg-parsechangelog give you all the changelog information, including version obviously but the point I want is to extract exactly and only version information. So i have to use it in conjunction with grep, sed

# dpkg-parsechangelog | grep Version | sed 's/Version: //g'

sed command is what I have learnt. sed uses with quotes, s/Version: looking for Version, /somecharacters to replace Version with somecharacters, and /g to look for string globally any occurrences will be replaced.

Here is the special case, I look for Version: but in replace I leave it empty so the next option /g goes right after /: sed 's/Version: //g'


Other examples in string manipulation

$ foo="1234567890"
$ echo -n $foo | tail -c 3

2/ If you want to know which process, program receiving, sending to which host and amount of data use these two commands below
On Recv: sudo watch -n .1 'netstat -tup | grep -E "^[t,u]cp[6]{0,1}" | sort -nr -k2'
On Send: sudo watch -n .1 'netstat -tup | grep -E "^[t,u]cp[6]{0,1}" | sort -nr -k3'

If you suspect that process is being triggered by another process: ps axf



3/ Want to know which file is opened in system, by which user and process id? # lsof



4/ Find out who is logged on and what they are doing
# w username
Ex: w root



5/ How to run specific sudo commands without a password

Use the NOPASSWD directive in /etc/sudoers file

Ex: add this line to /etc/sudoers file
user host = (root) NOPASSWD: /sbin/shutdown

user is the user you want to grant permission on host to run /sbin/shutdown command


Alternative way, you could add the line above to new file under /etc/sudoers.d/; ex: /etc/sudoers.d/shutdown

This way, you won't mess up the /etc/sudoers file, and separate different grants to commands/users



6/ Put your Bash function running in the background?

Use & at the end of your function call; it will be kept alive... forever! and you should set interval for your function to avoid processing consuming that might cause your system unstable

Ex:
function GetFreeMemInBackground {
          while true
          do
                    #do something here
                    sleep 60 # set interval in seconds for your function
          done
}

#then run your function in background
GetFreeMemInBackground &


You can also get the Process ID of your called function and keep it for use later, eg: you might want to kill it. The important point, you should get the process right after function call

GetFreeMemInBackground &

BackgroundFuncId=$!


Then kill it later:
kill $BackgroundFuncId > /dev/null 2>&1



7/ You want to get an output of a command and put into variable?
Simply put the command in "$()"

Ex: OUTPUT="$(ls -al)"

Quoting does matter to preserve multi-line values



8/ Want to know Free Memory in system
It is a combination of top, grep, awk

top -l 1 | grep PhysMem: | awk '{print $10}'

top -l 1: to get 1 sample of current system information

top -l 1 | grep PhysMem: 

then: top -l 1 | grep PhysMem: | awk '{print $10}': to print out the column 10th on the output of grep's output



9/ Want to have cool Terminal foreground and background color like this?

PS1 is all you need, PS1 defines how your terminal styles, colors, position of cursor, and information. You can get the output of command and put into PS1 as well. Like the snapshot above, you can see my cursor tells me current Free Memory in my system.

I have a function running in background, it runs every 60 seconds, get the free memory and put into temporary text file. Then PS1 will output that text file to cursor


Function to get Free Memory and put into text file

function GetFreeMemInBackground {

        while true

        do

                FreeMem="$(top -l 1 | grep PhysMem | awk '{print $6}')"

                echo $FreeMem > /Users/phuha/freemem.txt

                sleep 60

        done

}

GetFreeMemInBackground &

Another function to get Free Memory

function GetFreeMem {

        cat /Users/phuha/freemem.txt

}

Then PS1 gets the Free Memory info from function GetFreeMem

PS1='[FreeMem: `GetFreeMem`] '${userColor}'\u: '${pathColor}'\w'${themes[$(((RANDOM % 23)))]}'\n\$ '

10/ Install Java 8 on Debian

su -
echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main" | tee /etc/apt/sources.list.d/webupd8team-java.list
echo "deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main" | tee -a /etc/apt/sources.list.d/webupd8team-java.list
apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys EEA14886
apt-get update
apt-get install oracle-java8-installer
exit

Ref:
http://www.tecmint.com/command-line-tools-to-monitor-linux-performance/
http://www.binarytides.com/linux-commands-monitor-network/
http://www.cyberciti.biz/tips/top-linux-monitoring-tools.html
http://www.grymoire.com/Unix/Sed.html#uh-30
http://askubuntu.com/questions/159007/how-do-i-run-specific-sudo-commands-without-a-password
http://bash.cyberciti.biz/guide/Putting_functions_in_background
https://wiki.archlinux.org/index.php/Color_Bash_Prompt
http://www.webupd8.org/2014/03/how-to-install-oracle-java-8-in-debian.html
http://tldp.org/LDP/abs/html/string-manipulation.html

JackRabbit Access Control

While researching JackRabbit Access Control I struggled with a lot of new concepts then I found this article is very useful, one thing left for me is to read it patiently

Summary: TBC

The original source: http://wiki.apache.org/jackrabbit/AccessControl

The 'cached' version of source:

Basic privileges:

• jcr:read The privilege to retrieve a node and get its properties and their values.
• jcr:modifyProperties The privilege to create, remove and modify the values of the properties of a node.
• jcr:addChildNodes The privilege to create child nodes of a node.
• jcr:removeNode The privilege to remove a node.
• jcr:removeChildNodes The privilege to remove child nodes of a node.
• jcr:write An aggregate privilege that contains: jcr:read, jcr:modifyProperties, jcr:addChildNodes, jcr:removeNode, jcr:removeChildNodes
• jcr:all An aggregate privilege that contains all available permissions, including jcr:read, jcr:write and the advanced permssions.

Resource-based ACLs

Advantages:

• fully supported by the JCR API / specification
• very widely available ACL model (eg. file systems)
• simple resource inheritance
• default mechanism in Jackrabbit, no configuration needed

Disadvantages:

• cannot assign ACLs to non-existent nodes
• cumbersome when many users need un-groupable ACLs on a few resources (e.g. "subscriptions"), lots of ACL entries per resource
• permissions are stored right inside the content (can be cumbersome for backups, etc.)

Resource-based ACLs are stored per resource/node in a special child node rep:policy. This one will have a list of rep:GrantACE child nodes (usually named allow, allow0,...) for grant access control entries and rep:DenyACE child nodes (usually named deny, deny0,...) for deny access control entries.

Each ACE node has a rep:principalName STRING property pointing to the user or group this ACE belongs to, and a rep:privileges NAME multi-value property, containing all the privileges of this ACE.

Example for your examination (for both cases: Resource-based, and Principal-based)

AccessControlManager aMgr = session.getAccessControlManager();

// create a privilege set with jcr:all
Privilege[] privileges = new Privilege[] { aMgr.privilegeFromName(Privilege.JCR_ALL) };
AccessControlList acl;
try {
    // get first applicable policy (for nodes w/o a policy)
    acl = aMgr.getApplicablePolicies(path).nextAccessControlPolicy();
} catch (NoSuchElementException e) {
    // else node already has a policy, get that one
    acl = aMgr.getPolicies(path)[0];
}
// remove all existing entries
for (AccessControlEntry e : acl.getAccessControlEntries()) {
    acl.removeAccessControlEntry(e);
}
// add a new one for the special "everyone" principal
acl.addAccessControlEntry(EveryonePrincipal.getInstance(), privileges);

// the policy must be re-set
aMgr.setPolicy(path, acl);

// and the session must be saved for the changes to be applied
session.save();

Principal-based ACLs

Advantages:

• permissions can be assigned to non-existent nodes
• permissions are stored separately from the content (good for content replication, backup etc.)
• good for having many users with un-groupable ACLs (e.g. "subscriptions"); resources don't get filled up with ACL entries

Disadvantages:

• additional Jackrabbit API has to be used for setting ACLs
• modeling resource inheritance requires more ACLs than resource-based

An access control list (rep:ACL) is stored for each user and group (this is transparent, currently it's mirroring the users's home path at /rep:accesscontrol//rep:policy/). This consists of entries (rep:ACE), which are either allow (rep:GrantACE) or deny (rep:DenyACE) entries.

The rep:ACE nodetype (used by both resource- and principal-based ACLs) defines the following two properties for principal-based usage. These exact same names need to be used as restrictions when using the Jackrabbit API (JackrabbitAccessControlList.addEntry()):

• rep:nodePath defines the (base) path of a subtree the ACL applies to (mandatory, PATH property)
• rep:glob defines a glob pattern to restrict the subtree, both child nodes & properties (only a path matching is done) (optional, STRING property)

For the glob pattern, see http://jackrabbit.apache.org/api/2.2/org/apache/jackrabbit/core/security/authorization/GlobPattern.html (NodePath is the rep:nodePath and restriction is the rep:glob).

// usual entry point into the Jackrabbit API
JackrabbitSession js = (JackrabbitSession) session;

// get user/principal for whom to read/set ACLs

// Note: the ACL security API works using Java Principals as high-level abstraction and does not
// assume the users are actually stored in the JCR with the Jackrabbit UserManagement; an example
// are external users provided by a custom LoginModule via LDAP
PrincipalManager pMgr = js.getPrincipalManager();
Principal principal = pMgr.getPrincipal(session.getUserID());

// alternatively: get the current user as Authorizable from the user management
// (there is a one-to-one mapping between Authorizables and Principals)
User user = ((User) js.getUserManager().getAuthorizable(session.getUserID()));
Principal principal = user.getPrincipal();

// get the Jackrabbit access control manager
JackrabbitAccessControlManager acMgr = (JackrabbitAccessControlManager) session.getAccessControlManager();

JackrabbitAccessControlPolicy[] ps = acMgr.getPolicies(principal); // or getApplicablePolicies()
JackrabbitAccessControlList list = (JackrabbitAccessControlList) ps[0];

// list entries
JackrabbitAccessControlEntry[] entries = (JackrabbitAccessControlEntry[]) list.getAccessControlEntries();
JackrabbitAccessControlEntry entry = entries[0];

// remove entry
list.removeAccessControlEntry(entry);

// add entry
Privilege[] privileges = new Privileges[] { acMgr.privilegeFromName(Privilege.JCR_READ) };
Map restrictions = new HashMap();
ValueFactory vf = session.getValueFactory();
restrictions.put("rep:nodePath", vf.createValue("/some/path", PropertyType.PATH));
restrictions.put("rep:glob", vf.createValue("*"));
list.addEntry(principal, privileges, true /* allow or deny */, restrictions);

// reorder entries
list.orderBefore(entry, entry2);

// finally set policy again & save
acMgr.setPolicy(list.getPath(), list);
session.save();

How to add Observer to Magnolia CMS

JCR Node Manipulation Observation: When you want to monitor any actions taken on your JCR node, eg: Create, Update, Read, Delete... It's very easy to do it in Magnolia CMS

Magnolia has mechanism to manage observation, it's Observation module. You can find this module in Magnolia\Configuration app

In there, you can add your observer (class), and register any actions as your wish, workspace, node type, path, including sub-nodes? and other options. There are already many built-in observers in there for your reference

And your observation class should extend BaseRepositoryCommand, and look like below

Java Programming: Back to 2005 - Troubleshooting build issues

I didn't build any Java project far far away... from 2005, when I got my first job. Now, I'm coming back managing Java project... it's 2nd Java project that I have ever managed, this is very interesting project, and I decide to... play a role of developer

I try to build the project and I'm struggling with it!! So the motivation for this post is simple... taking notes of what I learn

I built my project, It was successful, without any errors message! I ran it, successful too... perfect! But it's weird, I cannot debug my project! I set the breakpoint, but it was not working!

My colleague, after few minutes troubleshooting, and we found the issues:

1/ The main project dependency pointing to module's version that is different from the current version that I have set in the module. So the main project is always pointing to the old Jar, not module's project

Although, before that we tried to 'MVN Clean', 'Update dependencies' for each project, it's still the same. The only way to solve the problem is going to main project POM and update the correct module's version

2/ I had learnt also that there were a Problem window in the Eclipse IDE, where I can see all the issues and I had to clean it up to make sure my project works correctly

3/ There were also Server window, where I can see my web project instance, and its dependent modules inside. If there were any problem with build, there should be missing modules inside that web instance. So you have to make sure all the dependent modules should be built and present in the web instance

Very first IonicFramework Hello World! app

Initially, I just want to test the new approach: recording my study/practice, and review it back in future and in case if I forget something the clip will help to remind me. Finally turn out, I think... why don't I publish it to my blog also? That's main reason you see this post :)

Okay, now we start!

Ref source: http://ionicframework.com/

1/ Install the Node.js

Download and install the Node/NPM

Do the Hello World demo

2/ Install the Cordova and Ionic

You gonna get error here! You need to change to ROOT permission

The installation is done, it seems OK!

I just finished the registration!

Um... it seems my project already get up! Now I need to run it only

But still would want to save these commands for further use

$ sudo npm install -g ionic cordova 

$ ionic start [project] 

$ cd [project] 

$ ionic login 

$ ionic upload

I would go on and on with the Tips here

- Just issue command: ionic serve in myApp folder. It's awesome!!!

- But that's server tided to localhost. I would run in public IP, and check if I able to access it from my mobile... wait a minute! Oh god! How do I change the default IP preset before. Great! just ionic help serve - you get all the helps you need

- Let run the app in public IP: ionic serve --address 192.168.80.34

- Now check the access from mobile.... NO THING HAPPEN!! I NEED TO ASK MY FRIEND FOR A HELP. He is able to login, but no content just and tags OR probably the firewall issue, okay talk about it later... move on!

- Um... seems like I just created another app neat inside the myApp lol :D

- Move on... this is interesting app, try this: ionic start myApp sidemenu, but the folder myApp already exist, try another name.. please

ionic start mySideMenuApp sidemenu

Oh My Godness!!!! I love it! You see!

- Now, let's try the other parts!!

Whooooopies......... ionic platform add ios, and then ionic build ios does all the stuffs!!! for you!

- Let check if that's true ;) um... that's not TRUE, you need to download the emulator yourself, let's do it! Btw, do we need changing to ROOT? probably NO :( - It's required for any new Node Package installation!!!!

Okay, go back to launch the new app.... um - I don't have xcode 6, that is required. I have now to install XCode 6...patient

It's not going to the right way now! um.... Okay we stop here! Hopefully, after upgrading to XCode6, it will work fine for me!

Cheers!!! See you next time! Pray for me, able to get it up and run!

THE INTERNET THINGS BUSINESS INDEX - A quiet revolution gathers pace - A report from The Economist Intelligence Unit

THE INTERNET THINGS BUSINESS INDEX
A quiet revolution gathers pace
A report from The Economist Intelligence Unit