Skip to content

add out of bounds check to cross entropy #588

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Feb 25, 2025
Merged

add out of bounds check to cross entropy #588

merged 3 commits into from
Feb 25, 2025

Conversation

shivam15s
Copy link
Collaborator

Summary

Same as title

Testing Done

  • Hardware Type:
  • run make test to ensure correctness
  • run make checkstyle to ensure code style
  • run make test-convergence to ensure convergence

lancerts
lancerts previously approved these changes Feb 25, 2025
View reviewed changes
@shivam15s shivam15s enabled auto-merge (squash) February 25, 2025 01:55
qingquansong
qingquansong approved these changes Feb 25, 2025
View reviewed changes
Copy link
Collaborator

@qingquansong qingquansong left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! How is the speed decrease due to this checking?

@shivam15s shivam15s merged commit 5156018 into main Feb 25, 2025
6 of 7 checks passed
@shivam15s shivam15s deleted the shisahni/fix_xent branch February 25, 2025 02:02
@shivam15s
Copy link
Collaborator Author

shivam15s commented Feb 25, 2025 •
edited
Loading

Smaller B*T suffer more with this extra checking.

B*T Orig Error Checking time diff % change
4096 0.3626239896 0.4262720048 0.06364801526 17.55206966
8192 0.5120959878 0.5754560232 0.06336003542 12.37268733
16384 0.952639997 1.017040014 0.06440001726 6.760163069
32768 1.683680058 1.753168106 0.06948804855 4.127152795
65536 3.290463924 3.35510397 0.06464004517 1.964466004
131072 7.458911896 7.528768063 0.06985616684 0.9365463464

Ideally want to do the assert in triton directly after loading a block

hebiao064 pushed a commit that referenced this pull request Feb 26, 2025
@shivam15s @lancerts @hebiao064
add out of bounds check to cross entropy (#588)
8464ba5 
## Summary
<!--- This is a required section; please describe the main purpose of
this proposed code change. --->
Same as title
<!---
## Details
This is an optional section; is there anything specific that reviewers
should be aware of?
--->

## Testing Done
<!--- This is a required section; please describe how this change was
tested. --->

<!-- 
Replace BLANK with your device type. For example, A100-80G-PCIe

Complete the following tasks before sending your PR, and replace `[ ]`
with
`[x]` to indicate you have done them. 
-->

- Hardware Type: <BLANK>
- [ ] run `make test` to ensure correctness
- [ ] run `make checkstyle` to ensure code style
- [ ] run `make test-convergence` to ensure convergence

---------

Co-authored-by: Shao Tang <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@qingquansong qingquansong qingquansong approved these changes

@lancerts lancerts lancerts left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@shivam15s @lancerts @qingquansong