I'm not familiar with ASIO, but I know that in network communication, integers are sent in "network byte order," which is most significant byte first, but Intel processors store integers least significant byte first.
I had that thought at first too, but on rereading his original message realized he did say "no matter what I set it to," and realized I must be thinking something way too simple to actually be the problem. :-)
Besides, ASIO takes care for the correct interpretation of the addres (including the right byte-order), you can set the IP&port in a "human-readable" form. According to the description of the OP, it seems that 11035 is the local port that the application opens for the outgoing connection - this explains why it's always the same.