Author Archives: admin

Installing kuberenets on vagrant

Just to get some knowledge and test products on it, it is vey useful to be have access to a cluster. One easy way is using Vagrant. Here is how I did it:

I am using a bare metal server running Centos 7.4.1708

  1. Software installation
    • Vagrant 2.0.3
    • VirtualBox 5.1.34r121010
  2. Setup vagrant
    1. Copy vagrant file from https://gist.github.com/smirzai/8724cb36bd7fcbe4d289b86586b70c5a
    2. vagrant up. You will have four machines named n1, n2, n3, n4. n1 will be the master node. Each machine will have two  network interfaces. Here the local is 192.168.0.0/16
  3. for each node prepare the machine:
    1. sudo apt-get update && sudo apt-get upgrade
    2. Install docker, kubectl, kubeadm, kubelet as described in installing kubeadm document
    3. swap off -a (This is not persistent. Should be done every time a machine restarts)
    4. The following changes should be enforced in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
      1. Environment=”KUBELET_EXTRA_ARGS=–node-ip=192.168.77.10 –enable-debugging-handlers=true  –cluster-dns=10.244.0.10 –cgroup-driver=cgroupfs –cluster-dns=10.244.0.10″
      2. The above –node-ip should be replaced with ip of the machine
      3. $KUBELET_CGROUP_ARGS should also be added to the line starting with ExecStart=/usr/bin/kubelet
      4. restart kubeadm by executing:
        1. systemctl daemon-reload
        2. systemctl restart kubelet
    5. Initialize the cluster:
      1. sudo kubeadm init –apiserver-advertise-address 192.168.77.10 –pod-network-cidr=10.244.0.0/16 –service-cidr=10.244.0.0/16
      2. The init command prints the statement for joining the nodes. Write it down somewhere.
      3. Update the kubectl config file:
        1. mkdir -p $HOME/.kube
          sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
          sudo chown $(id -u):$(id -g) $HOME/.kube/config
        2. or execute the following as roo:
          export KUBECONFIG=/etc/kubernetes/admin.conf
      4. Install flannel:
        1. download flannel creation yams file:
          wget https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml
        2. add “–iface=eth1” to the  flannel.yaml  in the following line:
          command: [ “/opt/bin/flanneld”, “–ip-masq”, “–kube-subnet-mgr” , “–iface=eth1”]
        3. create flannel cid network: kubectl -f flannel.yaml
        4. Now all nodes should be ready
        5. Join the n2, n3, n4 by executing the join command written down above in init step

 

Fast daily report bash script

I prefer to create a small asciidoc file as my daily report with minimum information about what I have done and some small notes and labels on what I have done.

Several times a day, I have to revert back to daily report and update it. To speed it up, I created this naive shell script.

It simply checks if the daily report file exists for today. If it does not exists, it copy one template, updates the date to today and return the full file in any case.

using this alias

alias edr=”emacs `~/opt/dr`”

I can create or update the daily report with 4 keystrokes.

Activiti docker image

Docker is a fantastic virtualization tool for running different images very fast and with minimum effort. There is a hub available with many pre-created ready to use images.

With docker it is matter of minutes to download and run images for Activiti explorer and rest Api.

It would be greatly helpful for people who want to play and evaluate the tool. Also for people who make demos and present Activiti.

First you need to install docker, if it is not already done. Installation is very straight forward. Binaries are available for nearly every major OS.

After installation, you need to pull the image:

docker pull smirzai/activiti-single-image

Downloading image may take some time, depending on your internet connection. After that you can easily run explorer or Rest-API.

Executing this command runs the explorer in the container:

sudo docker run -p 8080:8080  -t -i smirzai/activiti-single-image:latest explorer

After less than a minute, container will be running. You can access the explore using such an URL in your explorer:

http://localhost:8080/activiti-explorer

voilà ! you are there.

Similarly you can run Rest API, using such a command:

sudo docker run -p 8080:8080  -t -i smirzai/activiti-single-image:latest rest

Explorer is the default, if you omit the last parameter.

We are working on making it official out of my personal docker hub.

 

 

scip docker image

SCIP  seems to be a wonderful open source tool for solving constraint integer problems. It is the standard tools for many university courses. The problem is the installation is not trivial specially for pure mathematicians.

Docker comes to rescue. After straight forward installation of docker, you need only these few steps to run scip:

 

docker pull smirzai/scip

docker run -i smirzai/scip

scip

 

you are there. you can use scip now. The only catch is that you have to send your files to the container using  docker ps command.

 

PS: I was contacted by SCIP team and they kindly advised me to remove the docker image, as user does not need to accept license agreement with the current installation. I try to solve the problem with them and put the image back in docker hub. For now I have to remove it.

 

problems of confluence as document writing tool

Here is the list of problems I faced in using confluence for writing documents and then converting them to word or PDF.

1. Master/Slave versioning:

if you have a master document which consists of several child pages, each of them will have their own version history. consequently you cannot use versioning that comes with confluence for parent/child pages, unless you are happy with having several change history tables, with different version numbers in the final document.

2. Offline usage
You cannot work on your documents where you are on a train.

3. Loosing content
Ok, this is only one time experience. After half an hour working on documentation, I got a horrible error message “you have enough credentials” and all my work vanished.

4. source control
you cannot push your documents in source control. This means that you have to keep the source and the documentation in sync manually.

5. fine grained versioning
I don’t really want to increase the version each time I change a comma in the text. As far as I know there is no tool, to let me decide when version should be incremented.

6. hard editing
If I find a typo in my docs, which I often do, and I want to correct it, I need to scroll up to the top of the page, press edit button (this can be done faster by keyboard shortcut), scroll back and find the typo and correct it, then I need to scroll down to the bottom of the page and press save (For the last part I think ctrl-s is a keyboard shortcut). This can be cumbersome in long documents.

Activiti Maven Archetype

As activiti developer, It happens too often that one has to create sample test cases . Some boilerplate code is necessary even for the most basic test case to run. Things like sample config file, sample process model, test case.

To make it faster, Joram has created a “Activiti Unit Test template” which makes life much easier. Personally I have used it a lot. It does most of the job, but again you need to do some customization to fully use it. To say, if you use multiple copies, without change, you can import only one of them in Eclipse.

There are tools for scaffolding and generating which allow faster, more dynamic and customizable templates. As maven is already used, maven archetype seems to be the logical tool.

In the most recent version of activiti (version 5.17.1) , the  first activiti maven archetype contains the same code written by Joram, in a new shell.

The archetype will be available in standard activiti maven repository. This simple command, will create a unit test template.

mvn archetype:generate
-DarchetypeGroupId=org.activiti.archetypes
-DarchetypeArtifactId=activiti-archetype-unittest
-DarchetypeVersion=5.17.1-SNAPSHOT
-DgroupId=org.myGroup
-DartifactId=myArtifact

 

The first three parameters  specify the archetype to be used. GroupId and artifactId determines your customized groupId and artifactId. Packages will be created according to the given group Id.

As simple is that.

This is just the start. Lot’s of extensions are possible. Ideas like creating a template for a web application (one with angular.js and one with jsf), selecting database type and so on.

 

Real Multithreaded parallel execution in activiti

Here is a real scenario:

We are going to implement a  web service using activiti. So we should be synchronous and quick. In this scenario call comes from Camel endoint, but it is not relevant for this discussion and we can ignore it safely without loosing generality.

To conclude the service, we need to call 3 other external web services. External web services take random amount of time to return back the result. After all 3 web services are returned, we need to aggregate their returned results and return the result of aggregation as the return value of our web service.

This diagram shows the outline:

parallelSampleSerial

 

This works fine. As each step is synchronous, activiti executes the next step only when the previous one is returned. The returned values is stored as a process variable. At aggregation we have all the returned values and we can aggregate.

But if the web service calls are independent, we can run them in parallel and save time and resources. And here is where all the problems begin.

BPMN has already a parallel gateway available. The first temptation is  using that feature. something like this:

parallelSampleParallel

 

It looks nice on the paper, but we know that at activiti uses a single thread to run all the three paths. This means that at the end of the day, they are sequenced. There is no differences to the previous one, except that we cannot be sure about the order of the execution of web service calls. Not very promising !

What if we make the parallel services asynchronous? Ok, that is not easy as well. Here is a nice post from Tijs about how it could be done. It might be a little outdated though, thank to the new Async Job Executor starting from version 5.17.0.

With this approach, aggregate part is done properly. Activiti makes sure, the aggregate service is called only when all 3 web service calls are finished. good !

But the problem is that we need to return aggregated result as the return of web service. As the parallel tasks are asynchronous, the control returns to the caller immediately, without waiting for the external web service calls to return. So we don’t have the result yet. We need to wait and synchronize for the result to come. Usual way to do it is by locks and monitors in Java. Not exactly feasible here, as the the very same objects are not passed to the process. They are serialized and deserialized  via database.

To solve that problem, we can maintain a global static array of objects, and only pass the reference to the proper lock object to the activiti. In aggregation service, the static object is found and signaled with the help of passed index. Signaling causes the waiting thread to release and result to return back to web service.

Just as a rough proof of concept there I have created a  sample, available  in   parallelMultiThreaded in github.

In the sample MyUnitTest test class simulates the body of newly created web service call. Service1, Service2, Service3 simulate external web service calls, simply by sleeping. The executed model is the second one above with parallel gateway.

The static lock and condition arrays are defined as:

 public static Condition[] conditions = new Condition[1000];
 public static Lock[] locks = new Lock[1000];

 

index, is given a random number (0, not much random I confess) and passed to the activiti.
After starting the instance, main code goes to an unlimited wait loop, waiting for the corresponding object to be unlocked:

condition.await();

After all the external web services are executed, activiti executes aggregation task, which in turn signals the lock:

condition.signalAll();

The log files clearly show that everything goes fine:

02:35:16,817 [pool-1-thread-1] INFO  org.activiti.Service1  - Service 1 started, time = 70
02:35:16,817 [pool-1-thread-2] INFO  org.activiti.Service2  - Service 2 started, time = 70
02:35:16,817 [pool-1-thread-3] INFO  org.activiti.Service3  - Service 3 started, time = 70
02:35:17,318 [pool-1-thread-3] INFO  org.activiti.Service3  - Service 3 finished, time = 571
02:35:17,519 [pool-1-thread-2] INFO  org.activiti.Service2  - Service 2 finished, time = 772
02:35:17,819 [pool-1-thread-1] INFO  org.activiti.Service1  - Service 1 finished, time = 1072
02:35:17,859 [pool-1-thread-1] INFO  org.activiti.AggregateService  - All external ws calls returned. Try to unlock the lock. time = 1112
02:35:17,859 [main] INFO  org.activiti.MyUnitTest  - wait loop exited. Program finished. total taime spent = 1112

It is clear that tasks are running simultaneously  and aggregation and monitoring are done properly. Also there is no considerable delay for unlocking and signaling.

 

Ok, I am cheating. The time for sleeping are intentionally set to different values with proper distance. Otherwise contention will cause Asynchronous job executor, to retry. This adds several seconds to the whole process. But that is not the subject of this post.

The sample code is not considering optimistic locking problem as is described in the post above, neither it handles timeout in waiting for lock to be release,  but well shows the concept. So with this workaround, it is possible to use activiti parallel gateway for executing external web service really multi-threaded and respond the result synchronously.

But one may suggest to overlook activiti parallel gateways, and do the thread synchronization in a single Java delegate. That way it would be much  conciser and easier to implement. It is much faster, as it does need communicating via database. The cons is that you can not see the real flow in the diagram.

 

Obviously this only works when everything is running in a single jvm. It is possible to implement it in a cluster, but even the current implementation is not much practical to the long delay caused by asynchronous job picking mechanism

PS: Thank to Joram’s hint, I replaced the low level wait and notify to higher level ReenterantLock.

 

 

 

Exporting diagrams from Bizagi to Activiti Designer

Activiti comes with two modelers, eclipse based and web based. Activiti does not promise NO-CODE workflow and modelers are also designed having a developer as user in mind, specially the eclipse one. One can easily forget about eclipse based designer when trying to use it to discuss a business process with a domain expert or a business analyst. They need some more graphical tool than eclipse.

Bizagi is a robust clean free tool for such purposes. One can use it easily to draw a conceptual BPMN diagram and even simulate it in some extend.

But what if you decide to execute the model using Activiti ? In older versions, it was not possible to reuse the diagram available in Bizagi. The whole process diagram needed to be recreated in activiti from scratch, blank paper style. The most recent version (I am using version 2.5 for this test), supports exporting BPMN diagram to various formats including BPMN. This format is what activiti uses by default. So theoritically it should be possible to open the diagram created in Bizagi  inside eclipse based designer and edit it there.

But they do not funtion always as advertised. I did a short test to check how possible it is.

First of all, it is important to notice the two tools serve more or less different purposes. Activiti designer cover all the detailed required information to really execute the model by its own engine. Diagram in Bizagi is more a conceptual diagram and lack those details. So obviously in best case, the converted diagram is not executable. Only the base structure will be converted and all the details, class named, error codes, and so on should be added manually. Not ideal, but way much better that redrawing every thing from scratch.

Exported diagram can be opened in Eclipse designer and usually the basic patter nis readily visible. It is fine as long as you have not tried to edit something. The smallest change and save makes the whole disagram be unreadable the next time. The reason is that for any reason activiti does not understand the Collaboration semantics produced by Bizagi. Whose fault is it, should be checked and fixed. Activiti designer only uses it for graphical representation after all, no effect on the execution.
So one workaround is to delete the whole content of <Collaboration> tag as first step before trying to save the diagram.

The above trick makes the diagram editable in eclipse designer. Most of the commonly used BPMN elements are compatible from Bizagi to Activiti, but not all of them. I did a quick test on what is supported and what is not and here is the result:

UnSupported BPMN Elements:

  1. Subprocess
  2. CallActiviy
  3. Shell Task
  4. Camel Task
  5. Mail Tasl
  6. Error Start Event
  7. Cancel End Event (NPE, probably eclipse bug)
  8. Multi instance (Both Parallel or sequential)

Supported BPMN Elements:

  1. Start Events:
    • None Start Event
    • Timer Start Event
    • Message Start Event
    • Signal Start Event
  2. End Events:
    • None End Event
    • Error Event
    • Message End Event
    • Signal End Event
  3. Boundary Events:
    • Message boundary event
    • Error Boundary event
    • Signal Boundary event
    • Compensate Boundary event
    • Intemediate events::
    • Message boundary event
  4. Intemediate events:
    • Intermediate Message throwing event
    • Intermediate Message catching event
    • Intermediate Signal Catching event
    • Intermediate Signal Throwing event
    • Timer intermediate event
    • Signal throwing event
  5. Sequences:
    • Sequence Flow
    • Conditional Sequence Flows
  6. Gateways:
    • Exclusive gateway
    • Parallel gateway
    • Inclusive Gateway
    • Event based Gateway
  7. Tasks:
    • User Task
    • Script Task
    • Service Task
    • Receive Task
    • Manual task

Update in Web Service Mock demo

In follow up of my previous post about mocking web services, I rewrote the sample demo to better show the concept.
Not the there is a class multiplier that uses adder service. Now it is clearer to see that Multiplier is the class we want to test and not the adder.
The adder is mocked only to test the multiplier. I also removed the traditional way of creating a concrete web service for testing. It may confuse the reader.
The updated source is available in github: https://github.com/smirzai/webServiceMockDemo

There is also a available.