tag:blogger.com,1999:blog-73721502333098247722024-03-08T10:27:44.386+01:00Coding Obsessioncasting Java source code into a webAnonymoushttp://www.blogger.com/profile/00086213525958401372noreply@blogger.comBlogger6125tag:blogger.com,1999:blog-7372150233309824772.post-44716652983885913712012-03-20T12:23:00.001+01:002012-04-23T18:44:13.340+02:00Installing MongoDB on Fedora 16Needing to install MongoDB for some quick tests, I found lots of blogs/wikis about how to get started. All inaccurate regarding Fedora 16: no need at all to add additional repositories.<br />
<br />
<h2>Install</h2><br />
<div style="font-family: "Courier New",Courier,monospace;"><span style="font-size: large;">sudo yum install mongodb-server mongodb</span></div><br />
<h2>Start</h2><br />
<div style="font-family: "Courier New",Courier,monospace;"><span style="font-size: large;">sudo service mongod start</span></div><br />
<div>done!</div><br />
<h2>Optional: some more insight</h2><br />
<h3>Configuration</h3><div>File is at:</div><div>/etc/mongodb.conf</div><br />
<h3>Interesting options</h3><div># disallow table scans; if we do, I'd fix our code:</div><div>notablescan = true </div><div># enable HTTP interface</div><div>nohttpinterface = false</div><div># Verbose logging output.</div><div>verbose = true</div><br />
<h3>Where is the stuff?</h3><div>dbpath=/var/lib/mongodb</div><div>pidfilepath=/var/run/mongodb/mongodb.pid</div><div>logpath=/var/log/mongodb/mongodb.log</div><br />
<a href="http://localhost:28017/">Local console: http://localhost:28017/</a>
<h3>Have MongoDB start as a service</h3><div>systemctl enable mongod.service</div><br />Anonymoushttp://www.blogger.com/profile/00086213525958401372noreply@blogger.com0tag:blogger.com,1999:blog-7372150233309824772.post-67471765801316278172010-08-26T16:46:00.027+02:002010-12-01T00:56:01.302+01:00Using BoxGrinder meta-appliance to create custom EC2 AMIs<div style="text-align: justify;"><span class="Apple-style-span" style="font-family: inherit;">While I'm happy with BoxGrinder, I don't use it on a daily basis and keep forgetting the trivial instructions; so instead of stressing the project mantainers with the same questions I'll write the instructions here as a reference for myself and a tutorial for whoever might be interested.</span></div><div style="text-align: justify;"><span class="Apple-style-span" style="font-family: inherit;"><br />
</span></div><div style="text-align: justify;"><span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="font-size: x-large;">What is BoxGrinder?</span></span></div><div style="text-align: justify;"><span class="Apple-style-span" style="font-family: inherit;">It's a Ruby tool which creates virtual machines from simple definition files; these </span><i><span class="Apple-style-span" style="font-family: inherit;">Appliance Definition</span></i><span class="Apple-style-span" style="font-family: inherit;"> files are cloud-vendor and virtualization tecnology neutral, so you define your appliance once and with different plugins you have them running on different virtualization platforms. So while I'm experimenting my builds with </span><i><span class="Apple-style-span" style="font-family: inherit;">Amazon's EC2</span></i><span class="Apple-style-span" style="font-family: inherit;">, I know I'll be able to provide the same appliance to </span><i><span class="Apple-style-span" style="font-family: inherit;">VMWare</span></i><span class="Apple-style-span" style="font-family: inherit;"> systems, </span><i><span class="Apple-style-span" style="font-family: inherit;">VirtualBox</span></i><span class="Apple-style-span" style="font-family: inherit;">, </span><i><span class="Apple-style-span" style="font-family: inherit;">KVM</span></i><span class="Apple-style-span" style="font-family: inherit;">, even create </span><a href="https://jira.jboss.org/browse/BGBUILD-12"><span class="Apple-style-span" style="font-family: inherit;">bootable USB keys.</span></a></div><div style="text-align: justify;"><span class="Apple-style-span" style="font-family: inherit;"><br />
</span></div><div style="text-align: justify;"><span class="Apple-style-span" style="font-family: inherit;">It currently supports Fedora, Red Hat, CentOS operating systems and built images can be run on several targets; it also uploads the built image, in my case using EC2 it will register the new AMI for me.</span></div><div style="text-align: justify;"><span class="Apple-style-span" style="font-family: inherit;"><br />
</span></div><div style="text-align: justify;"><span class="Apple-style-span" style="font-family: inherit;">For a complete introduction, there's </span><a href="http://www.jboss.org/boxgrinder/video.html"><span class="Apple-style-span" style="font-family: inherit;"> a nice video on the official website</span></a><span class="Apple-style-span" style="font-family: inherit;">.</span></div><div style="text-align: justify;"><span class="Apple-style-span" style="font-family: inherit;"><br />
</span></div><div style="text-align: justify;"><span class="Apple-style-span" style="font-family: inherit;">Following instructions are collected from the official documentation which you can find </span><a href="http://community.jboss.org/en/boxgrinder"><span class="Apple-style-span" style="font-family: inherit;">here</span></a><span class="Apple-style-span" style="font-family: inherit;">, I only cherry-picked what I'm interested in for my own goals, so make sure to look at the full documentation in case you need more options or target different virtualization providers.</span></div><div style="text-align: justify;"><span class="Apple-style-span" style="font-family: inherit;"><br />
</span></div><div style="text-align: justify;"><span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="font-size: x-large;">Meta-Appliance</span></span></div><div style="text-align: justify;"><span class="Apple-style-span" style="font-family: inherit;">A meta-appliance is an appliance containing almost all what we need to build other appliances; so instead of installing all of the needed software locally, you can grab one from EC2's AMI catalogue and use it. I like this option as you'll end up having to upload your AMI to S3, so if you assemble it directly on Amazon's systems this process will be way quicker and spare your bandwith.</span></div><div style="text-align: justify;"><span class="Apple-style-span" style="font-family: inherit;">When selecting a meta-appliance, make sure you select the appropriate architecture: as of BoxGrinder 0.5.1 you need a 64bit meta-appliance to build 64bit AMIs, or a 32bit meta-appliance to build a 23bit AMI (this last limitation will likely be resolved soon as </span><a href="https://jira.jboss.org/browse/BGBUILD-46"><span class="Apple-style-span" style="font-family: inherit;">BGBUILD-46</span></a><span class="Apple-style-span" style="font-family: inherit;">).</span></div><div style="text-align: justify;"><span class="Apple-style-span" style="font-family: inherit;"><br />
</span></div><div style="text-align: justify;"><span class="Apple-style-span" style="font-family: inherit;">There we go:</span></div><div style="text-align: justify;"><span class="Apple-style-span" style="font-family: inherit;">start an instance of </span><b><i><span class="Apple-style-span" style="font-family: inherit;">ami-96db30ff</span><span class="Apple-style-span" style="font-style: normal; font-weight: normal;"><span class="Apple-style-span" style="font-family: inherit;"> and connect with your private key using SSH.</span></span></i></b></div><div style="text-align: justify;"><b><i><span class="Apple-style-span" style="font-style: normal; font-weight: normal;"><span class="Apple-style-span" style="font-family: inherit;"><br />
</span> </span></i></b></div><div style="text-align: justify;"><b><i><span class="Apple-style-span" style="font-style: normal; font-weight: normal;"><span class="Apple-style-span" style="font-family: inherit;">Then update the system, and install some tools:</span></span></i></b></div><div style="text-align: justify;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="color: #333333;"><br />
</span></span></div><div style="text-align: left;"><span class="Apple-style-span"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"></span></span></div><div style="text-align: left;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="color: orange;"><span class="Apple-style-span" style="background-color: white;">yum update</span></span></span></div><div style="text-align: left;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="color: orange;"><span class="Apple-style-span" style="background-color: white;">rpm -Uvh http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.noarch.rpm</span></span></span></div><div style="text-align: left;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="color: orange;"><span class="Apple-style-span" style="background-color: white;">yum install createrepo subversion</span></span></span></div><div style="text-align: left;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="color: orange;"><span class="Apple-style-span" style="background-color: white;">gem install boxgrinder-build boxgrinder-build-ec2-platform-plugin boxgrinder-build-s3-delivery-plugin boxgrinder-build-fedora-os-plugin</span></span></span></div><div style="text-align: left;"><span class="Apple-style-span"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"></span></span></div><div style="text-align: justify;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><br />
</span> </div><div style="text-align: justify;"><span class="Apple-style-span" style="font-family: inherit;">Ec2-ami-tools is a requirement to bundle the AMIs on S3, <i>createrepo</i> is useful to integrate own-built RPMs, and finally I use subversion to manage the appliance definition files.</span></div><div style="text-align: justify;"><span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="color: #333333;"><br />
</span></span></div><div style="text-align: justify;"><span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="color: #333333;">Now create some directories and the local RPM repositories:</span></span></div><div style="text-align: justify;"><span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="color: #333333;"><br />
</span></span></div><div style="text-align: justify;"><div style="text-align: left;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="color: orange;"><span class="Apple-style-span" style="background-color: white;">mkdir -p /opt/repo/RPMS/i386 /opt/repo/RPMS/noarch /opt/repo/RPMS/x86_64 ~/.boxgrinder/plugins</span></span></span></div><div style="text-align: left;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="color: orange;"><span class="Apple-style-span" style="background-color: white;">createrepo /opt/repo/RPMS/i386 && createrepo /opt/repo/RPMS/x86_64 && createrepo /opt/repo/RPMS/noarch</span></span></span></div><div style="text-align: left;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="color: #333333;"><br />
</span></span><span class="Apple-style-span" style="color: #333333;"> </span></div><div style="text-align: justify;"><span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="color: #333333;">Enter S3 credentials in the plugin configuration to be able to upload your AMI:</span></span></div><div style="text-align: left;"><span class="Apple-style-span" style="color: #333333;"><br />
</span><br />
<div style="text-align: left;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="color: orange;"><span class="Apple-style-span" style="background-color: white;">vi ~/.boxgrinder/plugins/s3</span></span></span></div><div style="text-align: left;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="color: #333333;"><br />
</span></span></div><div style="text-align: justify;"><span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="color: #333333;">The configuration looks like this, just omitting the passwords:</span></span></div><div style="text-align: justify;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="color: #333333;"><br />
</span></span><span class="Apple-style-span" style="color: #333333;"> </span></div><div style="text-align: left;"><div style="text-align: left;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="color: #333333;">access_key: xx</span></span></div><div style="text-align: left;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="color: #333333;">secret_access_key: xxx</span></span></div><div style="text-align: left;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="color: #333333;">bucket: scarlet-private-amis</span></span></div><div style="text-align: left;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="color: #333333;">account_number: 3441-4397-4060</span></span></div><div style="text-align: left;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="color: #333333;">cert_file: /root/cert.pem</span></span></div><div style="text-align: left;"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="color: #333333;">key_file: /root/soseaws.pem</span></span><span class="Apple-style-span" style="color: #333333;"><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"></span><br />
</span><span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="color: #333333;">You have to upload the cert.pem and soseaws.pem files, these are your private keys for accessing all of AWS services.</span></span><span class="Apple-style-span" style="color: #333333;"><br />
</span><span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="color: #333333;">Now get the appliance definitions:</span></span></div><div style="text-align: left;"><span class="Apple-style-span" style="color: #333333;"><span class="Apple-style-span" style="font-family: inherit;"></span><br />
</span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="color: orange;"><span class="Apple-style-span" style="background-color: white;">svn co https://dev.sourcesense.com/repos/dev/scarlet/trunk/appliance-build/appliances appliances --no-auth-cache --username s.grinovero</span></span></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="color: #333333;"><br />
</span></span><span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="color: #333333;">And create an AMI:</span></span><br />
<span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="color: #333333;"></span></span><span class="Apple-style-span" style="color: #333333;"><br />
</span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="color: orange;"><span class="Apple-style-span" style="background-color: white;">boxgrinder-build appliances/common-scarlet.appl -p ec2 -d ami</span></span></span><span class="Apple-style-span" style="color: #333333;"><br />
</span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="color: #333333;"><br />
</span></span><span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="color: #333333;">This will end as:</span></span><span class="Apple-style-span" style="color: #333333;"><br />
<br />
</span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="background-color: white;"><span class="Apple-style-span" style="color: #444444;">I, [2010-08-26T09:14:52.070877 #18565] INFO -- : Bundling AMI...</span></span></span><span class="Apple-style-span" style="background-color: white;"><span class="Apple-style-span" style="color: #444444;"><br />
</span></span><span class="Apple-style-span" style="color: #444444;"> </span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="background-color: white;"><span class="Apple-style-span" style="color: #444444;">I, [2010-08-26T09:16:44.546519 #18565] INFO -- : Bundling AMI finished.</span></span></span><span class="Apple-style-span" style="background-color: white;"><span class="Apple-style-span" style="color: #444444;"><br />
</span></span><span class="Apple-style-span" style="color: #444444;"> </span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="background-color: white;"><span class="Apple-style-span" style="color: #444444;">I, [2010-08-26T09:16:44.547023 #18565] INFO -- : Uploading common-scarlet AMI to bucket 'scarlet-private-amis'...</span></span></span><span class="Apple-style-span" style="background-color: white;"><span class="Apple-style-span" style="color: #444444;"><br />
</span></span><span class="Apple-style-span" style="color: #444444;"> </span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="background-color: white;"><span class="Apple-style-span" style="color: #444444;">I, [2010-08-26T09:17:43.906981 #18565] INFO -- : Image successfully registered under id: ami-723ed41b.</span></span></span><span class="Apple-style-span" style="background-color: white;"><span class="Apple-style-span" style="color: orange;"><br />
</span></span> <span class="Apple-style-span"><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="color: #333333;"><br />
</span></span></span><span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="color: #333333;">Done! Now take a note of the registered AMI id and use it to start N-copies of it.</span></span><br />
<span class="Apple-style-span" style="font-family: inherit;"></span><br />
<span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="font-size: x-large;">Appliance Definition files</span></span><br />
<span class="Apple-style-span" style="font-family: inherit;">Again, to see all the options read the Stormgrind documentation, anyway an appliance definition looks like:</span><br />
<span class="Apple-style-span" style="font-family: inherit;"></span><br />
<span class="Apple-style-span" style="font-family: inherit;"><span class="Apple-style-span" style="font-size: large;">Example definition</span></span><br />
<div><br />
</div><div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b>name: common-scarlet</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b>summary: Base AMI definition common to all scarlet AMIs</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b>version: 2</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b>release: 1</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b>os:</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> name: fedora</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> version: "13"</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> password: xxx</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b>hardware:</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> cpus: 2</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> memory: 512</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> partitions:</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> /:</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> size: 3</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b>packages:</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> includes:</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> - bash</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> - yum</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> - vim-minimal</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> - openssh-server</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> - chkconfig</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> - acpid</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> - dhclient</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> - openssh-clients</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> - mc</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> - subversion</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> - mutt</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> - rsync</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> - s3cmd</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b>repos:</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> - name: "local-noarch"</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> baseurl: "file:///opt/repo/RPMS/noarch"</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> ephemeral: true</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> - name: "local-#ARCH#"</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> baseurl: "file:///opt/repo/RPMS/#ARCH#"</b></span></div><div><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> ephemeral: true</b></span></div></div></div></div></div></div><div style="text-align: justify;"><br />
Note the pointers to the local RPM repository, there you can add RPM packages which you want to be included in the built appliance, in my case my own application which is not available in the default Fedora repositories.<br />
<br />
<span class="Apple-style-span" style="font-size: large;">Hudson Definition</span><br />
to have a nice Fedora 13 with all latest updates running latest Hudson, just have BoxGrinder grind this definition:<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b>name: hudson-scarlet</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b>summary: Hudson instance to run builds of Scarlet</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b>version: 2</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b>release: 1</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b>os:</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> name: fedora</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> version: 13</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> password: yyy</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b>hardware:</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> cpus: 2</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> memory: 512</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> partitions:</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> /:</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> size: 3</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> /var/lib/hudson/:</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> size: 50</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b>appliances:</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> - common-scarlet</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b>packages:</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> includes:</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> - hudson</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> - java-1.6.0-openjdk</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b>repos:</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> - name: "hudson-repo"</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> baseurl: "http://hudson-ci.org/redhat/"</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b>post:</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> base:</b></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><b> - "/sbin/chkconfig --level 345 hudson on"</b></span><br />
<div><br />
</div><div><span class="Apple-style-span" style="font-family: inherit;">Note the inheritance to the appliance of the previous example: this appliance will contain also the parent's packages, then an additional repository is enabled to download Hudson and a large partitions is dedicated to it.</span></div><div><span class="Apple-style-span" style="font-family: inherit;">Finally, a post-build script is started which will make sure Hudson is started, you won't even need to login to the machine to configure it.</span></div></div>Anonymoushttp://www.blogger.com/profile/00086213525958401372noreply@blogger.com0tag:blogger.com,1999:blog-7372150233309824772.post-50603851095574721502009-12-08T00:24:00.004+01:002009-12-08T16:20:02.045+01:00Blogging now also at in.relation.toFrom today I'm also writing on the very cool <a href="http://in.relation.to/">in.relation.to</a> blog.<br />
This is the collective blog of experts from the Seam and Hibernate teams - I'm reading it since years - and am honored they invited me to write about my contribution to Hibernate Search.<br />
It's amazing how these very esteemed developers welcome contributions and are open to any kind of discussion.<br />
<br />
Recently I saw some very sad statements<b> </b>about the JBoss community not being truly open, or not being meritocratic; I think that people who believe that either didn't ever try to really contribute, or had met the wrong person at the wrong moment: as all communities, they are big and made of humans.<br />
When I started willing to contribute I wasn't an expert at all, still I was welcomed for my interest in the project and I always got - and still get - polite answers even to my most silly questions and doubts. After the traditional couple of patches were accepted, I slowly began feeling as part of a team. I might have been lucky, but luck has endured as every single person I keep meeting in these groups is at the same time very kind, smart and helpful. Just keep in mind they're all very busy: an answer could take some time.<br />
<br />
So today I wrote <a href="http://in.relation.to/Bloggers/HibernateSearch32FastIndexRebuild">this post</a> about Hibernate Search's new MassIndexer: read it, comment about it, make use of it! Then ask for improvements and join the fun :-)Anonymoushttp://www.blogger.com/profile/00086213525958401372noreply@blogger.com0tag:blogger.com,1999:blog-7372150233309824772.post-83888056136572929212009-09-08T01:07:00.016+02:002010-12-01T00:57:19.801+01:00What is Hibernate Search?<span style="font-weight: bold;"></span><br />
<div style="text-align: justify;">I'm getting this question relatively often, so I think that existing information online is making too many assumptions, or is too practical; I'll try to fill this gap with a very basic introduction.<br />
Hibernate Search is an open source Java project which integrates Hibernate with Lucene; both libraries have proven themselves extremely useful and are stable and of widespread use; in practice many projects face the need to use both.<br />
Unfortunately the String world of Lucene is quite different than the Hibernate world and every project trying to integrate both is doomed to face the same problems, to rewrite more or less the same glue, to have more code to maintain because of their own bugs, or because of API changes in one or both frameworks.<br />
<br />
<span style="font-weight: bold;">Lucene</span><br />
Is an Apache library which provides full-text capabilities: you create an index (in memory, on filesystem, in a database,...) and then you can search this index on keywords, phrases, boolean queries, etc..<br />
The results are commonly returned by relevance, so the best matching documents are returned first (think of it as a web search engine like google); the main point is that you have full control about how your items are parsed before entering the string world of the index, to choose which information is important for your business, how you define the matching rules. It is very fast and generally considered stable, still new features are constantly added.<br />
Being extremely flexible, working directly with Lucene is like programming "low level" so often applications introduce a separation layer to standardize the way it is used across an application, thus hiding some of the flexibility and possibly introducing some helpers.<br />
<br />
<span style="font-weight: bold;">Hibernate</span><br />
the aim of this very successful open source project is to simplify the interaction between the application and the database; technically it's and <span style="font-style: italic;">Object-Relational Mapping</span> service; you'll find plenty of information and tutorials about it on the web. The important point to introduce <span style="font-style: italic;">Hibernate Search</span> is that it makes you use POJOs to define the domain model of your application, annotating them to define the mapping to the database, and provides good APIs and even an object-oriented query language to interact with the database, all nicely fitted in a transactional world.<br />
<br />
<span style="font-weight: bold;">Hibernate Search<br />
</span>Hibernate Search is built on top of Lucene, like Hibernate is built on top of your SQL database. As Hibernate maps POJOs to tables, Hibernate Search maps them with to Lucene's index introducing a new set of annotations. The interesting point here is that you annotate with both families of annotations the same entities, and when you make an Hibernate query to the database or a Lucene query to the index, you'll get Hibernate managed entities in both cases. You define your domain model - which is unique - and how it maps to the database and to the index. When you make changes to your data the service will update both database and index at transaction commit.<br />
The API to run and paginate queries is an extension of Hibernate's (and JPA) API, so the changes in an application to introduce full-text capabilities are minimal.<br />
When using Lucene the code ususally gets quite verbose, like when defining Analyzers or Filters; with Hibernate Search you can define these declaratively and reuse them by name. Last but not least it makes use of several performance improving tricks, like: sharing file buffers across concurrent reading sessions, caching filter results, batching index changes, clustering solutions. All nice capabilities which you don't need to know, but they are there in case you'll need them.<br />
<br />
<span style="font-weight: bold;">Flexibility<br />
<span style="font-weight: bold;"></span></span>Even being a simplifying layer between the application and Lucene, it won't hide any advanced feature but provide tools to make use of them. Developers can customize all aspects: from defining custom bridges for your types<span style="font-weight: bold;"><span style="font-weight: bold;"></span> </span>up to replacing/extending whole parts of the framework. Each mayor component can be replaced with custom code: define your own index storage strategy by creating a custom <span style="font-style: italic;">DirectoryProvider</span>, use your own <span style="font-style: italic;">LockManager</span>, create a new <span style="font-style: italic;">IndexShardingStrategy</span>, fine-tune all performance settings which Lucene exposes. If you're still missing something, you're free to change the code and submit patches.<br />
<br />
<span style="font-weight: bold;">Websites:<br />
</span><a href="http://search.hibernate.org/">Hibernate Search - website</a><span style="font-weight: bold;"><br />
</span><a href="https://forum.hibernate.org/viewforum.php?f=9">Hibernate Search - forums</a><span style="font-weight: bold;"><br />
</span><span style="font-size: 100%;"><a href="http://lucene.apache.org/java/docs/index.html">Lucene's Java implementation website</a></span><br />
<br />
<span style="font-weight: bold;">Books:</span><br />
<span style="font-size: 100%;"><a href="http://www.manning.com/bernard/">Hibernate Search in Action</a><br />
</span><a href="http://www.manning.com/bauer2/"><span style="font-size: 100%;">Java Persistence with Hibernate</span></a><br />
<span style="font-size: 100%;"><a href="http://www.manning.com/hatcher3/">Lucene in Action, Second Edition</a></span> </div>Anonymoushttp://www.blogger.com/profile/00086213525958401372noreply@blogger.com5tag:blogger.com,1999:blog-7372150233309824772.post-56474025818410315352008-09-28T15:26:00.003+02:002008-09-29T00:20:01.054+02:00Ideas on Hibernate SearchI'm reading the preview of the upcoming excellent <a href="http://www.manning.com/bernard/">Hibernate Search in Action</a> and I am getting inspiration about some improvements I would like to implement, in particular:<br /><br /><span style="font-weight: bold;">Multiple backends</span><br />As each index has it's own DirectoryProvider why not also it's own backend?<br /><ul><li>needing an entity to be indexed "sync" while others "async".</li><li>wanting an index using JMS, another async / local.</li><li>using different JMS queues without a selector.</li></ul>The implementation is not hard at all, I'll have to move some classes from the refactored org.hibernate.search.backend.impl.lucene (I'm working on it already) to org.hibernate.search.backend; probably most work will be about discussing which is the best and simplest way for users to configure them.<br /><br /><span style="font-weight: bold;">A scalability improving ReaderProvider</span><br />The ReaderProviderS have all to guarantee the returned indexreader is absolutely updated; however this doesn't make much sense in "async" mode, as the user would probably prefer a slightly outdated reader traded for some extra throughput.<br />In my opinion most users sill use the "async" mode for most entities, in particular if we enable different backends as in my experience there usually are a feq entities which need "sync" mode.<br />I was thinking about implementing a new ReaderProvider which could "wrap" another implementation (for flexibility) and then periodically retrieve a new IndexReader from the wrapped one in a configurable time period.<br />So two initialization arguments: backing implementation (class or name), frequency period (ms).<br />This way if the wrapped ReaderProvider is a plain NotSharedReaderProvider the index would be reopened each other X ms:<br /><ul><li>drawback: potentially opening more than needed under low load.</li></ul><ul><li>advantage: the actual rate is controlled even under high load.</li></ul>Additionally if it is wrapping a smarter implementation like <span style="font-style: italic;">shared</span> or <span style="font-style: italic;">shared-segments</span> the drawback will degenerate in just some useless new file checks, instead of really reading all data.<br /><br />I am actually sorry I didn't have this idea earlier, as I think the implementation is trivial but needs good docs and explanation... too late for inclusion in the book?<br /><br /><span style="font-weight: bold;">Automatic Sharding strategies</span><br />As Emmanuel explains in the book the IdHashShardingStrategy strategy provided with H.Search is more like a "demo" strategy as the most interesting strategies depend on the user needs.<br />It occurs me the tips he is giving could apply very well on an entity having an Enum property, in which case a great IndexShardingStrategy could be generated automatically and "autoconfigured" as we know already the number of elements and are guaranteed they have different names.. good for index names postfixes or something like that.<br />Just add an annotation to the field, something like<br /><span style="font-weight: bold;">@ShardDiscriminator</span>.<br /><br /><span style="font-weight: bold;">Improved Filters exploiting improved Sharding...</span><br />Having created such an optimal IndexShardingStrategy will further enable the code to create a special Filter to be used during searches, which is capable of improving Search performance by selecting the correct Index to search in (avoiding the search in all indexes).<br />When adding a new value the SearchFactory must be restarted anyway, so the fact you can't add a new Enum value dynamically should not be a limit: as you should not anyway, I actually like the fact that you can't. Looks like java compiler enforces correct Hibernate Search configuration and usage...<br /><br />The same logic could be applied to any String field or @ManyToOne pointing to an entity<br />whose number will not change and has a way to be transformed in a unique index indentifier (require a toString or have it implement an interface?); We could start supporting the Enum and see how good it is.Anonymoushttp://www.blogger.com/profile/00086213525958401372noreply@blogger.com0tag:blogger.com,1999:blog-7372150233309824772.post-14435542771670072092007-07-23T14:19:00.000+02:002008-09-28T23:23:42.900+02:00Updating to JBoss 4.2.1<span style="font-size:100%;">I'm going to update an enterprise application from using JBoss 4.0.5 to JBoss 4.2 and writing here the memorandum of most important steps and considerations, as asked by friends and my boss.</span><br /><span style="font-size:130%;">Why updating?</span><span style="font-weight: bold;"><br /></span><div style="text-align: left;"><ul><li><span style="font-size:100%;">Version 4.2 fixes some bugs which were affecting our application.</span></li><li><span style="font-size:100%;">Red Hat <a href="http://www.redhat.com/about/news/prarchive/2007/application_platform.html">announced</a> they are going to use 4.2 in their Enterprise Application Platform, so they are committed to support this version for a long time.</span></li><li><span style="font-size:100%;">I had already updated some libraries of JBoss 4.05 to use some features we where needing; the new version comes with the same updated libraries so I think they will be better compatible than my souped-up, unsupported version.</span></li></ul><span style="font-size:130%;">Technologies</span><span style="font-weight: bold;"><br /></span>The application to migrate is a JavaEE web application, developed in Eclipse and using these technologies:<br /><ul><li><a href="http://labs.jboss.com/jbossseam">Seam</a> 1.2.1 is the integrating framework.<br /></li><li>Faceletes for page design<br /></li><li><a href="http://www.hibernate.org/">Hibernate</a> 3.2 : core, search, annotations and entitymanager.<br /></li><li><a href="http://lucene.apache.org/java/docs/">Lucene</a> 2.2 for fast full-text searching: both using hibernate search and custom code.<br /></li><li><a href="http://jtds.sourceforge.net/">JTDS</a> as JDBC driver.</li><li>SQL Server 2000 as Database on a windows server<br /></li><li>Fedora linux 6 for the webserver.</li><li>"some" JSF implementation...</li></ul>We were using the myfaces JSF implementation, but the JBoss and Seam people now recommend using the Sun reference implementation, and this comes bundled as default in the new application server. You have two options:<br /><ol><li>keep the myfaces JSF implementation.<br /></li><li>update to Sun's RI JSF implementation.</li></ol>We had no particular need to keep myfaces, and being the second option recommended by the JBoss team, we are actually going to see how painless it is to switch implementations.<br /><br />So now we can begin our<br /><span style="font-weight: bold;font-size:180%;" >Migration checklist to JBoss 4.2.1 and Sun's JSF</span><br /><span style="font-size:100%;">from Jboss 4.0.5 and myfaces.</span></div>In <span style="font-weight: bold;">application.xml</span> remove the following modules from application.xml:<br /><blockquote><pre><module><br /><java>el-api.jar</java><br /></module><br /><module><br /><java>el-ri.jar</java><br /></module></pre></blockquote>and add this one instead:<br /><pre><blockquote><module><br /><java>commons-collections-3.1.jar</java><br /></module></blockquote></pre>You will <b>need to remove</b> and add the relative jars to the root of your ear. The commons-collections-3.1 is needed by Ajax4jsf; Jboss ships now with a different version.<span style="font-weight: bold;"><br /></span><p></p><p>In <span style="font-weight: bold;">faces-config.xml</span> add<br /></p><pre><application><br /><el-resolver>org.jboss.seam.jsf.SeamELResolver</el-resolver><br /><message-bundle>messages</message-bundle><br /></application></pre>and update the headers to:<br /><pre><faces-config version="1.2"<br />xmlns="http://java.sun.com/xml/ns/javaee"<br />xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"<br />xsi:schemaLocation="http://java.sun.com/xml/ns/javaee<br />http://java.sun.com/xml/ns/javaee/web-facesconfig_1_2.xsd"></pre><p></p><p>In <span style="font-weight: bold;">web.xml</span> remove this listener:<br /></p><pre>org.apache.myfaces.webapp.StartupServletContextListener</pre>and all other references to myface's classes. No new listener should be needed.<br />When using tomahawk you may like to keep some org.apache.myfaces context parameters, as it should work on Sun's RI but uses the parameters named in myface's style.<p></p>You may like to update the header of <span style="font-weight: bold;">jboss-app.xml</span>:<br /><pre><?xml version="1.0" encoding="UTF-8"?><br /><!DOCTYPE jboss-app<br /> PUBLIC "-//JBoss//DTD J2EE Application 4.2//EN"<br /> "http://www.jboss.org/j2ee/dtd/jboss-app_4_2.dtd"><br /></pre><span style="font-size:180%;">Problems & some fixes<br /></span><pre style="font-weight: bold;"><span style="font-size:130%;">Content-Type</span></pre>Using myfaces the rendered pages have "Content-Type: text/html;" as reported by Firefox's plugin "web developer", using Sun's RI the content type is now "Content-Type: application/xhtml+xml;".<br />Technically the second one should be better, as discussed by <a href="http://www.weiqigao.com/blog/2004/01/20/the_courage_to_serve_xml.html">here</a>, but it brings some issues:<br /><ol><li>Even with same HTML and CSS the pages could look-like different.</li><li>Internet explorer (upto 6) doesn't like "xhtml+xml"</li><li>Some redirects won't work.<br /></li></ol> The good news is that you will get firefox to do a full check of your pages, so maybe you'll find some errors faster. To get the content as text/html you define the contentType attribute in your views: <pre><f:view contentType="text/html"...</pre>Still this leaves a problem with Seam's PDF rendering: there's no way to modify the content type of the redirect servlet that brings you from the page to the pdf download link, I'm going to see if I can get apache to force the "correct" contentType served to clients.Anonymoushttp://www.blogger.com/profile/00086213525958401372noreply@blogger.com0