Why doesn't concatLayer in Deep Learning Toolbox concatenate the 'T' dimension?

1 view (last 30 days)
Hello,
While implementing a ViT transformer in Matlab, I found at that the concatLayer does not concatenate over the T dimension. This is needed to concatenate the class token with patch tokens, since the natural representation is CBT with C corresponding to features, B to batch and T to token within a batch (this is also the canonical representation in the attention function).
It's possible to work around this by hacking to e.g. SCB, but then other problems pop up which also need to be hacked around.
Thx

Accepted Answer

Ben
Ben on 14 Mar 2023
You can create a layer that concatenates on the T dimension with functionLayer
sequenceCatLayer = functionLayer(@(x,y) cat(3,x,y));
This will work in dlnetwork to concatenate two CBT dlarray-s.
Since you're concatenating the class token, it might also be worth considering creating a custom layer that has the class token embedding as a Learnable property, and performs the concatenation in the predict method.
  3 Comments
Catalytic
Catalytic on 23 Mar 2023
Edited: Catalytic on 23 Mar 2023
@John Smith - Since Ben's answer yielded a solution for you, you should hit the Accept this Answer button, and likewise with other answers you might not have accepted.
Artem Lensky
Artem Lensky on 19 Aug 2023
Are there any plans to make concatenationLayer support concatetnation along the T dimension?

Sign in to comment.

More Answers (0)

Categories

Find more on Image Data Workflows in Help Center and File Exchange

Products


Release

R2022b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!