Friday, 29 November 2013

Changing the encoding of a currently edited file in NetBeans is not possible. In NetBeans all files of a project should have the same encoding. TO convert all files of a project to same encoding use another tool.

Project specific file encoding

To set a project specific encoding, right click on a project (in Project View) and choose "Properties" from the context menu. Under "Sources" you can set the encoding in the dropdown field "Encoding".

Default file encoding

To set default file encoding in NetBeans, go to NetBeans installation directory and edit "etc/netbeans.conf".
Add the option "-J-Dfile.encoding=UTF-8" to "netbeans_default_options":
netbeans_default_options="-J-Dfile.encoding=UTF-8 ..."
Restart NetBeans and the default encoding should be UTF-8. Check this under "Help - About":

To show more infos beside the filenames shown in the Project View you can define special "Subversion Status Labels" formats.
Go to: "Tools - Options - Miscellaneous - Versioning - Subversion - Subversion Status Labels - Format"-field and enter the format you want, e.g. "{author} {folder} {revision}"
Possible info labels can be seen under "Add Variable..."



Monday, 25 November 2013

Logging is an essential cross cutting aspect in (Java) applications. A widely used logging library is log4j (Homepage), another project of the Apache Group. To abstract the logging implementation from your application it is common use to put SLF4j (Homepage) as logging facade in front of the implementation (here log4j).

Installation

To use log4j and slf4j in your Maven-project you need this dependencies:
<dependency>
  <groupId>org.slf4j</groupId>
  <artifactId>slf4j-api</artifactId>
  <version>1.6.4</version>
</dependency>
<dependency>
  <groupId>org.slf4j</groupId>
  <artifactId>slf4j-log4j12</artifactId>
  <version>1.6.4</version>
</dependency>
<dependency>
  <groupId>log4j</groupId>
  <artifactId>log4j</artifactId>
  <version>1.2.17</version>
</dependency>

Configuration

The logging configuration is done for the underlying implementation. For log4j the config file "log4j.xml" (or log4j.properties) has to be in the root of the classpath. If not, specify the path to the config file through the system property "log4j.configuration" (this is the preferred way to specify the default initialization file):
-Dlog4j.configuration=path/to/your/config.xml
The configuration manual is here.

Output Format

Output format is defined using PatternLayout configuration.
Example:
<layout class="org.apache.log4j.PatternLayout">
  <param name="ConversionPattern" value="[%d{ISO8601} %-5p] %-25c{1} (%-8t) > %m%n"/>
</layout>
Description:
Pattern Description
%d{ISO8601} timestamp in format "yyyy-MM-dd HH:mm:ss,SSS", e.g. "1999-11-27 15:49:37,459".
%-5p Loglevel (left aligned, min. 5 characters long)
%-25c{1} category of the logging-event (left aligned. min. 25 characters long, only last classpath element (class name / last package) printed in full length, parent packages shortend to one character)
%-8t thread-name (left aligned, min. 8 characters long)
%m%n logging message and line break
Example output:
[2013-09-18 08:36:48,802 INFO ] UserServiceImpl            (http-8080-12) > try to login user 'heino'

Appenders (targets for the output)

File

Logging usually goes to a file on the local filesystem. Example configuration of a FileAppender (%X{...} is explained in detail in this article):
<appender name="myFileAppender" class="org.apache.log4j.RollingFileAppender">
  <param name="File" value="/var/log/webapp/my-webapp.log" />
  <param name="append" value="false" />
  <param name="MaxFileSize" value="5MB" />
  <param name="MaxBackupIndex" value="20" />
  <layout class="org.apache.log4j.PatternLayout">
    <param name="ConversionPattern" value="[%d{ISO8601} %-5p] [%X{username} / %.5X{sessionID}...] %-25c{1} (%-8t) > %m%n" />
  </layout>
</appender>

Email

For production it is useful to get Exceptions immediately reported to the developers. This can be achieved by using the SMTPAppender.
<appender name="mail-errors" class="org.apache.log4j.net.SMTPAppender">
  <param name="SMTPHost" value="..." />
  <param name="From" value="noreply@..." />
  <param name="To" value="email1@...,email2@..." />
  <param name="Subject" value="MyWebapp Exception in production" />
  <param name="BufferSize" value="1" />
  <param name="threshold" value="error" />
  <layout class="org.apache.log4j.PatternLayout">
    <param name="ConversionPattern" value="%d %-5p [%X{username} / %.5X{sessionID}...] [%t] %l > %m%n" />
  </layout>
</appender>

Usage

Logging can be implemented explicitely in the code or using AOP (see this article about Spring AOP logging). Manual logging is done this way:
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
 
public class XYZ {
  private static final Logger logger = LoggerFactory.getLogger(XYZ.class);
  ...
  public void someMethod(String arg) {
    logger.info("someMethod called with argument {}", arg);
  }
}
Note: There are discussions about the member "logger" if it should be in uppercase (because it is static) or not. After reading some discussions http://stackoverflow.com/questions/3842823/should-logger-be-private-static-or-not and http://stackoverflow.com/questions/1417190/should-a-static-final-logger-be-declared-in-upper-case I changed my previous behavior (using uppercase "LOGGER") and started to use lowercase ("logger")... What are your best practices?

Thursday, 21 November 2013

When running a webapplication in production you need the ability to retrace all actions an user has done.
Especially is this interesting when an error/exception occured. In this article you will learn to add the session-id and the username of the logged in user to every single line logged into the logfile.

Log4j's MDC (Mapped Diagnostic Context)

Log4j offers a Thread-local Key-Value-Store to store information for the lifetime of a request. It is called MDC (Mapped Diagnostic Context) (see
http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/MDC.html).Values stored in this context can be referenced over their key in a log4j-configuration to be used in the output format.

Store Session-ID and username using a Servlet-Filter

A webapp filter has access to the request and the related session. We implement a filter to retrieve the session-id and store the value in the MDC:
com.pairoo.frontend.webapp.filter

import java.io.IOException;

import javax.servlet.Filter;
import javax.servlet.FilterChain;
import javax.servlet.FilterConfig;
import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpSession;

import org.slf4j.MDC;

public class LogSessionIdFilter implements Filter {
  @Override
  public void destroy() {
  }

  @Override
  public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException,
    ServletException {
    HttpServletRequest httpRequest = (HttpServletRequest) request;
    HttpSession session = httpRequest.getSession();
    if (session != null) {
      MDC.put("sessionID", session.getId());
    }
    chain.doFilter(request, response);
  }

  @Override
  public void init(FilterConfig arg0) throws ServletException {
  }
}
As long as there is no session, the MDC will have "sessionID" not set. The output (reference to the sessionID-value) in the log-line will be emtpy.
The same way other informations (like the username) can be retrieved in the "doFilter" method and be stored under a key. As this is specific for every webapp, you should debug into the session to find out how to get more session-specific objects (like the authenticated user and his username).

Register the filter in the web.xml

Add this filter and his mapping to your web.xml (note: if it depends on other filters, locate it after them to be sure that they are executed before):
<filter>
  <filter-name>logSessionIdFilter</filter-name>
  <filter-class>com.pairoo.frontend.webapp.filter.LogSessionIdFilter</filter-class>
</filter>
...
<filter-mapping>
  <filter-name>logSessionIdFilter</filter-name>
  <url-pattern>/*</url-pattern>
</filter-mapping>
    
<filter-mapping>
  <filter-name>wicketfilter</filter-name>
  <url-pattern>/*</url-pattern>
</filter-mapping> 

Add Session-ID value to Log-Format

You can reference MDC values by their key (e.g. "sessionID") with "%X{key}" (e.g. "%X{sessionID}") and insert at any location inside the ConversionPattern:
<appender name="FILE" class="org.apache.log4j.rolling.RollingFileAppender">
  ...
  <layout class="org.apache.log4j.PatternLayout">
    <param name="ConversionPattern" value="[%d{ISO8601}] %5p [%X{sessionID}] [%t][%x] %c.%M - %m%n" />
  </layout>
</appender>
If you just want to show let's say the first 5 characters of the session-Id you can shorten it like this (using "%.5", see article about logging here):
...
    <param name="ConversionPattern" value="[%d{ISO8601} %-5p] [%X{username} / %.5X{sessionID}...] %-25c{1} (%-8t) > %m%n"/>
...
The example additionally added an example for logging the current username (that certainly has to be set in the MDC before).

Example-Output

The above ConversionPattern logs the session-id after the logging level (here: INFO) in brackets. It is not only logged in webapp-layer, but in all log outputs of all layers. So it is easy (using grep for one session id) to find out, what happened for one user. If you log out the username (like we did) you get one line that links the username to the session-id:
[2013-11-21 21:59:03,068]  INFO [jk4n0ulydj7a1onb4lig7fe8a] [9413464@qtp-9651822-0][] com.pairoo.business.services.impl.UserAccountServiceImpl.login - tries to login: username = adam
[2013-11-21 21:59:03,095]  INFO [jk4n0ulydj7a1onb4lig7fe8a] [9413464@qtp-9651822-0][] com.pairoo.frontend.webapp.wicket.WicketWebSession.login - user 'adam' logged in
That's it!

Wednesday, 20 November 2013

In this series of articles we will have a look into the dark corners of todays productive Java Legacy applications. Yes, you read right: Java Legacy.
In my definition I see Java lagcy code like this:
"Java legacy code is code that should no longer written the way it was written because it has been replaced for good reasons with more secure, performant, less boilerplate language constructs or frameworks."

Legacy JDBC SQL code

One of such legacy code appearances is SQL-statement construction by string concatenation.
Here is an excerpt of legacy code I had to fix (inserted "..." to anonymize it; note: left all concatenation lines to highlight how long the code really was! and to show project managers how much effort is necessary to fix legacy code, even if it is only an update statement):
String sql = "UPDATE " + ... + " SET ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += ... + "=" + ... + ", ";
        sql += " WHERE " + ... + "=" + ...;
Statement stmt = null;
ResultSet rs = null;
try {
  stmt = connection.createStatement();
  int cnt = stmt.executeUpdate(sql);
  if (cnt != 1) {
    throw new DAOException("update failed");
  }
} catch (SQLException e) {
  throw new DAOException(e);
} finally {
  try {
    if (stmt != null) {
      stmt.close();
    }
    if (rs != null) {
      rs.close();
    }
  } catch (Exception e) {
    log.error("Exception in DB resources cleanup");
  }
}
Yeah, you counted right: it's 65 columns!
As we all know: concatenating SQL statements by concatenating a pure string, that also contains user input as values is a security risk (SQL injection: inject a single quot to terminate SQL followed by a drop table...).

Alternatives

So what technique to use as replacement?
You could use
  • a PreparedStatement instead (see Java Tutorial) where you get placeholders for the values that are safe against SQL injection or
  • if the application already is "Spring"ified: use NamedParameterJdbcTemplate from Spring
  • ... other data access framework (Hibernate, iBatis, ...)
Using plain PreparedStatement has the disadvantage of a flood of "?" placeholders that you have to fill over their position index in the statement (1 to 65 !).
That's exactly where NamedParameterJdbcTemplate jumps in: it does not use "?" placeholders but "named parameters", so it is easy to fill the values into the statement without caring about error prone position hopping.

Spring NamedParameterJdbcTemplate

Using Spring's NamedParameterJdbcTemplate requires a DataSource.

public NamedParameterJdbcTemplate(DataSource dataSource) {
  Assert.notNull(dataSource, "DataSource must not be null");
  this.classicJdbcTemplate = new JdbcTemplate(dataSource);
}

As you can see in the legacy code above: we just have an existing connection (it is a class member in the example project) that we use without closing it after the update.
So how to get a javax.sql.DataSource if we just have a java.sql.Connection-instance?
Fortunately Spring offers a DataSource implementation that exactly fits our needs: SingleConnectionDataSource
It just needs a connection in the constructor.

Renovated code using  NamedParameterJdbcTemplate

The replacement for the legacy code finally looks like this (inserted "..." to anonymize it):

StringBuilder sql = new StringBuilder();
sql.append("UPDATE ").append(...).append(" SET ");
sql.append(...).append(...);
sql.append(...).append("= :nachname,");
sql.append(...).append("= :vorname,");
...
sql.append(...).append("= :geburtsdatum,");
...
sql.append("LATEST_UPDATE").append("= (select systimestamp from dual),");
...
sql.append(" WHERE ").append(...).append(...);

MapSqlParameterSource map = new MapSqlParameterSource();
...
map.addValue("...", ...);
... 
map.addValue("geburtsdatum", person.getGeburtsdatum());
map.addValue("nachname", person.getNachname());
map.addValue("vorname", getStringParam(model.getVorname()));

try {
  SingleConnectionDataSource dataSource = new SingleConnectionDataSource(connection, true);
  NamedParameterJdbcTemplate namedParameterJdbcTemplate = new NamedParameterJdbcTemplate(dataSource);
  int cnt = namedParameterJdbcTemplate.update(sql.toString(), map);

  if (cnt != 1) {
    throw new DAOException("update failed");
  }
} catch (DataAccessException | SQLException e) {
  throw new DAOException(e);
} finally {
  
}
Note:
  • String concatentation replaced with StringBuilder
  • Plain SQL statement replaced with NamedParameterJdbcTemplate (uses PreparedStatement under the hood)
  • values are set over MapSqlParameterSource (or direct in SQL statement, see LATEST_UPDATE)
  • reusing existing connection with SingleConnectionDataSource (minimal invasive: not introducing new connection-handling logic)
  • Exception handling: added DataAccessException handling
  • finally clause: removed Statement and ResultSet cleanup as stmt and rs no longer used
The legacy code purely worked with String-values in opposite to NamedParameterJdbcTemplate that works with typed values (Long, Boolean, String, Date, ...). So some effort also went into changing the logic for setting the right values...

That's it! The new code to update the data of the object is now SQL-injection save, more readable and maintainable.

Insert, delete, query

If you just want to execute a statement (like insert, delete, ...; above code did explicitely an "update"-call), the usage of the namedParameterJdbcTemplate is as follows:
Boolean b = namedParameterJdbcTemplate.execute(sql, map, new PreparedStatementCallback<Boolean>() {
  @Override
  public Boolean doInPreparedStatement(PreparedStatement ps) throws SQLException, DataAccessException {
    return ps.execute();
  }
});
A query that returns a ResultSet is as follows:
ResultSet rs = namedParameterJdbcTemplate.execute(sql, map, new PreparedStatementCallback<ResultSet>() {
  @Override
  public ResultSet doInPreparedStatement(PreparedStatement ps) throws SQLException, DataAccessException {
    return ps.executeQuery();
  }
});

Saturday, 9 November 2013

If you are older than 40 (like me), it may happen that you own some LPs or a whole collection of them near your old stereo.
Throwing away... never!
But, what a pitty, there are no digital copies you could buy, and to be serious: I payed for them with my own hard earned money back then when I was 16 and do not want to pay again!
So I want to digitize them to MP3, or as an true open source fan to OGG format. What do I need for this?
  1. A turntable (LP player): as mentioned before - I still have my old stereo: a Pioneer rack with a PL-880 turntable
  2. An amplifier: I have the Pioneer Stereo Amplifier A-330
  3. A Cinch-to-Jack-plug (Klinkenstecker) to connect the amplifier to your computer's microphone/line-in socket.
  4. A recording software. I used audacity under Linux.

Hardware wiring

And here are the steps for doing the wiring of the hardware:
  1. Check that your turntable is correctly connected to the amplifier (phone sockets)
  2. Connect the Cinch-end of the cable to a record socket of the amplifier: I used the record-sockets of "Tape 2" device:
    The Cinch-end connected to Rec-sockets of Tape 2 device
    (cable on the right going up on the photo)


  3. Connect the jack plug end to the jack socket of line-in at your computer's soundcard. I connected it to the line-in of my notebook (Lenovo Thinkpad T-500):
That's it for the hardware part.

Software Settings

  1. This is the most important information you will red in this post:
    Set the sound volume of your speakers, headphone and master to zero!
    (I didn't do this in the beginning and the result were distorted recordings, no matter how I tuned the volume of the amplifier or my soundcard's capture device!). So this is my settings of the playback devices (ALSA):
    All tuned down to zero.
  2. The capture settings: I just tuned the volume of "Microphone" up to a level, where the recording amplitude in audacity peaked to a maximum of -20 dB (see picture below):
    Audacity during a recording: note the amplitude (red bars on the top). The track's amplitude seems to be very small, but this ok.

Recording

When settings are fine tuned (do some test recordings to fine tune volumes) the recording is pretty simple:
  1. Place your vinyl LP on the turntable, needle hovered above the starting position before first track
  2. Press red record-button in Audacity to start recording
  3. Lower down your needle softly
  4. Wait until the whole side of the LP is recorded (e.g. until the needle automatically is lift)
  5. Press yellow stop-button in Audacity to stop recording
  6. Save audacity project for later sound processing

Sound processing

The recording is now a raw record that we want to improve a little bit: it still has pops and scratches to be removed.
Read the Wiki of Audacity about this. In short:
  • mark the track range between the pop when the needle gets in contact with the LP and the beginning of the first song (a range with no music)
  • select "Effect - Noise Removal..." from the menu
  • click on "Get Noise Profile" (audacity gets the noise profile from the "silent" market track section), the window disappears
  • mark the whole track (Strg+A) and select "Effect - Noise Removal..." again
  • click now on "Ok" to start noise removal for the whole track
That's it for what I did. This may not be the only step for audiophil classic music experts, but it is good enough for my heavy metal ;-).

Cutting

Now you have one track with all songs of a side of a LP.
The next step is to cut them to single songs.
  1. Zoom out of the track to see the pause gap between the song to be marked and the next song ("View - Zoom out")
  2. Mark the song to be cut out (with some silence in the beginning and the end)
  3. Save the region in the target format you want: "File - Export Selection..."
    Note: I set the OGG quality to 10 under "Options...", just out of a feeling (I don't want to loose quality, because I am not going to redo the recordings again...).Update: 7 is enough (around 1MB/s)
  4. In the next dialog window edit the metadata of the track (the values are prefilled when you do this again for the next song)
  5. After around 10 seconds the OGG-file is saved. Ready to Rock!
  6. Repeat this procedure for every song, every side and every LP...
Good luck!

Oh: and do not forget to take a picture of the cover, to be used by the different MP3-players: