In composite, we need to modify the reference to set <binding.direct> .
For SOA-OSB, we should set the transport protocol to sb protocol of proxy service.
For OSB-SOA, we should set the transport protocol to soa-direct of business service.
For the features of SOA-Direct, see OSB-SOA Direct transport
For how to implement a global transaction, we can refer to Edwin blog.
Global Transactions and Quality of Service in OSB