Thursday 5 December 2013

Boost program options

I've been using boost program_options, and I've grown to like it. However, after setting up the options for a few programs, I felt the need to minimize the code repetition involved.

My first idea was to put the options in a configuration file and read it at start-up. But I found the add_options() syntax ill-suited for this, and it would require a great deal of work to get all the possible combinations working (required options, default values, etc), even assuming I'd limit all options to value<string>; and that's without going into multivalue options (e.g., value<vector<string>>). So, I've decided to settle for good enough - I've divided the work required to configure the options in two classes.

The first class is a class template, AppOptions, and it takes care of the generic stuff, i.e., that which remains unchanged in every app.

template <typename SpecificOptions>
class AppOptions
{
public:
    AppOptions(int argc, char* const argv[], char const* opTitle);

 
    void PrintHelp() const { std::cout << desc << std::endl; }
    const SpecificOptions& GetOptions() const { return so; }
    bool HasShownHelp() const { return shownHelp; }
private:
    template <typename SpecOpt, typename CharT>
        friend std::basic_ostream<CharT>&
        operator<<(std::basic_ostream<CharT>& os, 
        AppOptions<SpecOpt> const& obj);

 
    bool GotRequestForHelp(po::variables_map const& vm, 
        SpecificOptions const& so) const
        { return vm.count(so.GetHelpOption()); }

 
    bool shownHelp;
    po::options_description desc;
    po::variables_map vm;
    SpecificOptions so;
};


The SpecificOptions template parameter will contain all the different options/logic for each application. We expose it through GetOptions(), because the app will need it to access the actual option values.

opTitle is the caption shown on the help message's first line. shownHelp is set to true when the user requests the "help message", i.e., a list of the app's options.

template <typename SpecOpt, typename CharT>
friend std::basic_ostream<CharT>&
operator<<(std::basic_ostream<CharT>& os, AppOptions<SpecOpt> const& obj)
{
    os << obj.so;
    return os;
}


operator<<() is rather basic, not much to see here, except to note that this requires that SpecificOptions also overloads operator<<().

template <typename SpecificOptions>
AppOptions<SpecificOptions>::AppOptions(
    int argc, char* const argv[], char const* optTitle) :
    shownHelp{false}, desc{optTitle}
{
    so.DefineOptions(desc);
    po::store(po::parse_command_line(argc, argv, desc), vm);

 
    if (GotRequestForHelp(vm, so))
    {
        shownHelp = true;
        PrintHelp();
    }
    else
    {
        try
        {
            po::notify(vm);
            so.Validate();
        }
        catch (po::required_option& ro)
        {
            BOOST_THROW_EXCEPTION(ConfigRequiredOptionMissing() <<
                config_error_string("Required option missing: " +
                ro.get_option_name()));

        }
    }
}


The ctor is where everything is set up. After calling SpecificOptions::DefineOptions() we get the atual command line arguments passed to our app. I borrowed an idea from a Stack Overflow article, to check for the help option before calling program_options::notify(); this call performs validations (e.g., required options), and it makes no sense to perform those validations if the user just wants to see our help message and option list (there's no reason why I couldn't make GotRequestForHelp() a free function, but I'll leave it as a member, for now).

Also, if that's all the user wants to see, it makes no sense to continue processing and we have to let the app know that. At first, I thought about throwing an exception from the ctor. In the end, I settled for setting a flag and allowing the app to check it.

You'll note the so.Validate() call. I first considered custom validators, thinking it could make for a neater design. However, the docs mention these as a way of customizing parsing/conversion, not actually performing validation, and I didn't like the design I found in the examples that used them (with regards to validation requirements). Also, I figured such a design would make it more difficult to validate sets of related options. And then I noticed the example that performs validations of related options ($BOOST_DIR/libs/program_options/example/real.cpp) also uses auxiliary functions to perform such validation, so that settled it.

The exception thrown is defined in its own header, which contains other exceptions, and it's a simple boost exception.

So much for the generic part. What about the specific?

Let's use an example from one of my apps (renamed, to protect critical trade secrets):

class ExampleOptions
{
public:
    void DefineOptions(po::options_description& desc);
    void Validate();
    std::string GetHelpOption() const { return "help"; }
    std::string GetOperation() const { return operation; }
private:
    friend std::ostream& operator<<(
        std::ostream& os, ExampleOptions const& obj);
 

    std::string operation;
};

 
ostream& operator<<(ostream& os, ExampleOptions const& obj)
{
    os << "Operação: " << obj.operation;

    return os;
}

 
void ExampleOptions::DefineOptions(options_description& desc)
{
    desc.add_options()
        ("help,h", "Mensagem de ajuda")
        ("oper,o", value<string>(&operation)->required(),
            "Operação a efectuar. C - Operação de crédito;\n"
            "P - Operação de parametrização")
    ;
}

 
void ExampleOptions::Validate()
{
    if (operation == "P" || operation == "p" ||
        operation == "C" || operation == "c")
    {
        return;
    }

 
    BOOST_THROW_EXCEPTION(ConfigInvalidOption() << 
        config_error_string("Operação desconhecida: " + operation));
}

Nothing tricky here, just a couple of notes.
  1. We only defined operator<<() for ostream, not for wostream, even though the operator<<() we defined for AppOptions is prepared for both.
  2. We're outputting plenty of non-ANSI characters with little concern for locale issues. Obviously, this causes problems, particularly on a Windows console. On my next post I'll show the solution I came up with to deal with this.
And how do we use it?

int main(int argc, char *argv[])
{
    AppOptions<ExampleOptions> appOpt{argc, argv, "Opções de Exemplo"};
  
    if (appOpt.HasShownHelp())
    {
        return 0;
    }
  
    cout << appOpt << endl;

 
    string const op = appOpt.GetOptions().GetOperation();
    if ((op == "P") || (op == "p"))
    {
    ...
}


Pretty simple, and all we have to repeat for each app is the code that is actually different. I still itch to tackle that "options in a config file" design, but I'll leave that to another day.

Thursday 28 November 2013

Every now and then one must emerge to let the world know one's alive

So...

1. The puzzle mentioned in my previous post is solved. I've increased the +/- 4M lines to 16M, and the difference from pass-by-value to pass-by-reference increased to a little over 1 min. So, there's a difference, but it's irrelevant. This was also a good example of something I've read before, that the compiler takes certain liberties when optimizing classes it knows - such as making calls directly to the implementation of standard containers, instead of using the usual interface. In this case, instead of using std map's copy ctor, it directly invoked the underlying red-black tree copy implementation.

So, the song remains the same - measure first, and optimize only what's required.

2. I've upgraded to Qt Creator 2.8.1/Qt 5.1.1, and I've rebuilt all the libs to work with mingw's gcc 4.8. I've also created scripts to automate building boost and ICU. While developing the boost script (in perl), I've come across a peculiar problem, but this time I've decided to work around it (half the work is in perl, the other half in a DOS batch script), instead of trying to figure out the cause. Life's too short, and all that jazz...

3. Watching Alex Stepanov's A9 lectures made me realize I was familiar only with C++'s OO paradigm, and that I had to diversify. So, I've began approaching my designs in a different fashion, applying generic programming whenever possible and resisting the urge to turn almost everything into an object. I'll address this in future posts, hopefully.

4. My libssh2 + boost asio work has been deployed. Not in the grandiose manner I envisioned, filled with pools of ssh connections and a multitude of threads, but in a more modest setting - a command line utility that reads a few files and runs a list of commands on a remote server. It's indeed a humble beginning, but it has already proved its success, and it will allow me another shot at something that got me stuck - how to apply logging to it (actually, for this utility, I #defined out all logging, because I wasn't happy with it).

I believe I got stuck because I tried to solve the problem of applying logging to reusable code, which is more complex than the problem I'll address now - how to apply logging from the point of view of an app (i.e., the code the uses the reusable code). I hope this different - and simpler - point of view will prove more productive.

Sunday 6 October 2013

g++ optimization mysteries

It's been a long time. Work has gone into overdrive, courtesy of a huge scope project. And not only has my spare time taken a hit, but, more importantly, there have been too many days when I get home and all I can do is pick up a book, or pick up my guitar, or watch a nice TV show with my wife (we're quite partial to Jamie Oliver shows on 24 Kitchen).

Still, I haven't abandoned C++, quite the opposite. I've been working on a few utilities to process files, and I've been experimenting with different options, and measuring each option's performance.

The last one I've tried left me with an interesting - and, so far, unsolved - puzzle.

We have a file where each line contains a field in a form; a whole form can be eight or nine lines (depending on whether all the form fields were filled). I've created a program that reads this file and outputs each form's fields on one line; something like this:

while (getline(...))
{
    // TOKENIZE THE LINE
    // GET THE FIELD'S DESCRIPTION AND VALUE
    // USE THE FIELD'S DESCRIPTION AS A MAP INDEX AND 
    //     STORE THE VALUE IN THE MAP
    // IF WE'VE READ ALL THE FIELDS, PRINT THE FORM
}


"Printing the form" means outputting the map to cout, which I did like this on my first version:

void OutputForm(CobForm& form)
{
    cout << form["dist"] << ";" << form["conc"] << ";" << 
        form["tlm"] << ";" << form["freg"] << ";" << 
        form["status"] << ";" << form["ID"] << ";" << 
        form["dt"] << ";" << form["result"] << ";" << 
        form["tipo"] << endl;
}


Actually, my first version had a CobForm const&, which the compiler prompty rejected, reminding me - probably in a slightly amused tone - that map's operator[] isn't const.

So, I ran this against the 3.8M lines input file, and it took +/- 1:32 on my PC; I ran it 10 times, and the variation was minimal, less than 1 second.

Then, I've decided to change it to pass-by-value:

void OutputForm(CobForm form)
{
    cout << form["dist"] << ";" << form["conc"] << ";" << 
        form["tlm"] << ";" << form["freg"] << ";" << 
        form["status"] << ";" << form["ID"] << ";" << 
        form["dt"] << ";" << form["result"] << ";" << 
        form["tipo"] << endl;
}

Another 10 runs, and no noticeable difference, +/- 1:33, again with variation below 1 second between runs.

At first, I thought maybe this function was inlined. However, nm disagreed:


00401b70 T OutputForm(std::map<std::string, std::string, 
    std::less<std::string>,
    std::allocator<std::pair<std::string const, 
    std::string> > >)


Yes, I was building in release mode. I added this to my Qt Creator project file, to keep the linker from removing the symbols:

QMAKE_LFLAGS_RELEASE -= -Wl,-s
 

Next, I've decided to look at the generated assembly. I'm far from an expert, but I thought "No harm in taking a look, right?"

First, pass by reference. This is what the compiler generated:


119             OutputForm(form);
0x40588e  <+0x19cd>         movl   $0x8,-0x1e8(%ebp)
0x405898  <+0x19d7>         lea    -0x150(%ebp),%ecx
0x40589e  <+0x19dd>         mov    %ecx,(%esp)
0x4058a1  <+0x19e0>         call   0x401b70  
                                <OutputForm(std::map<std::string, 
                                std::string, std::less<std::string>,
                                std::allocator<std::pair<
                                std::string const, std::string> > >&)>


With the following addresses:


form: 0x28fcf8
ebp:  0x28fe48
esp:  0x28fc10


So, we start by moving 8 into 0x28fc60 (-0x1e8(%ebp)), although I couldn't figure out why. Then, we move form's address into ecx, move that into the address pointed by esp (setting up the call argument), and we call OutputForm(). In OutputForm(), I was able to verify that form's address was 0x28fcf8.

Next, I did the same thing for the pass by value version. Which got me this:


119             OutputForm(form);
0x40593d  <+0x1a5c>         movl   $0x9,-0x218(%ebp)
0x405947  <+0x1a66>         lea    -0x150(%ebp),%ecx
0x40594d  <+0x1a6c>         mov    %ecx,(%esp)
0x405950  <+0x1a6f>         call   0x401b70 
                                <OutputForm(std::map<std::string, 
                                std::string, std::less<std::string>,
                                std::allocator<std::pair<
                                std::string const, std::string> > >)>

form: 0x28fcc8
ebp:  0x28fe48
esp:  0x28fbe0



Now, first thing I noticed is - no copy ctor. We're passing by value, but there's no copy ctor. Just to make sure, I made a debug build, and there it was, in all its splendor:


0x402262  <+0x00f6>         call   0x40a3e8 <std::map<std::string, 
                                std::string, std::less<std::string>,
                                std::allocator<std::pair< 
                                std::string const, std::string> > >::
                                map(std::map<std::string, std::string, 
                                std::less<std::string>,
                                std::allocator<std::pair< 
                                std::string const, std::string> > > const&)>


Then, I thought - maybe the compiler's decided it can perform a "sort-of-pass-by-ref" optimization. However, form's address was 0x28fcc8, and the address set up as an argument was -0x150(%ebp), i.e., 0x28fcf8. And, as soon as I stepped into OutputForm(), I was able to confirm that form's address in its scope was, indeed, 0x28fcf8.

Another thought occured me - perhaps it's some sort of alias. Maybe there's been no string allocation at all. So, I went back to the previous scope, and checked the addresses of form[0]:


form[0].first:  0x783618
form[0].second: 0x78361c


Then, I went into OutputForm() again, and the addresses were different:


form[0].first:  0x783c18
form[0].second: 0x783c1c

So, even though I can find no trace of a copy ctor, the fact is the addresses are different at each scope. And, for now, I have no idea what exactly is going on here.

However, this just shows, once again, the importance of measuring before "optimizing".


Sunday 1 September 2013

Boost.Locale, ICU and Mingw

I've been doing a few small command-line programs, and needed to output text like this on a Windows terminal: "Opção inválida".

That led me to the Interesting World of Internationalization. Which is a lot more Interesting on Windows than on Linux, I must say. This should work, right?

UINT oldcodepage = GetConsoleOutputCP();
SetConsoleOutputCP(CP_UTF8);
cout << "Olá\n" << endl;
SetConsoleOutputCP(oldcodepage);


In fact, not only did it not work reliably on all the Win7 machines I tried, but the endl wasn't outputted (hence, the "\n"). So, I've decided I needed something more powerful, like, say, ICU. And, fortunately, the size of the ICU DLLs would not be a problem.

Building ICU was easy. I just fired this on an msys terminal:

runConfigureICU MinGW --prefix=<DBG_PATH> --enable-debug --disable-release
make
make install

make clean

runConfigureICU MinGW --prefix=<REL_PATH> --disable-debug --enable-release
make
make install


No, I didn't bother with the recommended C++ changes, I'll look at it next time.

Next, building Boost. First thing was finding out what did I need to do to let Boost.Locale (and, as I found out later, also Boost.Regex) know that I had ICU available.

When invoking b2, I had to pass -sICU_PATH=<PATH_TO_ICU_HOME>. However, since I wanted to use different ICU versions for debug and release, that meant I couldn't just build Boost as I usually do, with --build-type=complete. Instead, I went for something like this:

b2 -sICU_PATH=<ICU_DBG_PATH> --prefix=<BOOST_INSTALL_DIR> 
    --build-dir=<BOOST_BUILD_DIR> toolset=gcc link=static,shared 
    runtime-link=shared threading=single,multi variant=debug 
    install > boost_install.log 2>&1

And I took a look at boost_install.log, as Boost started running its tests, including checking for ICU. And that was a good thing, because I spotted this early on:

- has_icu builds           : no

Boost was complaining about icui18n.dll and icudata.dll. And, sure enough, looking at my ICU lib folder, neither was anywhere to be found.

So, I went to Boost.Locale's (and Boost.Regex's) Jamfile.v2, looking for these dependencies. The first thing I noticed was that these dependencies didn't apply to MSVC. Actually, MSVC's dependencies included DLLs that I had on my system, so I renamed icui18n to icuin, and removed icudata. I also had to remove a couple of this_is_an_invalid_library_names, and finally I got this:

- has_icu builds           : yes

which made me quite happy. So, after Boost debug got built, I ran this little thingie to check that everything was OK:

#include "boost/locale.hpp"
using namespace boost::locale;

#include <algorithm>
using std::for_each;
#include <iostream>
using std::cout;
using std::endl;
#include <locale>
#include <string>
using std::string;

   
int main(int /*argc*/, char */*argv*/[])
{
    localization_backend_manager lbm = 
        localization_backend_manager::global();
    auto s = lbm.get_all_backends();
    for_each(s.begin(), s.end(), [](string& x){ cout << x << endl; });

    generator gen;
    std::locale loc = gen("");
    cout << boost::locale::conv::between("Olá", "cp850", 
        std::use_facet<boost::locale::info>(loc).encoding()) << endl;
} 


This outputted

icu
winapi
std
Olá


This not only shows that Boost.Locale was built with ICU support (line 1), but also that the conversions are working (line 4).

Next: Testing this on a few machines.